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— ;  This  paper  is  a  survey  of  existing  literature  describing 
software  quality  assurance  and  an  iniepth  evaluation  of  both 
selected  industry  quality  assurance  functions  and  the  Fleet 
Material  Support  Office  (FMSO)  Quality  Assurance  Division. 
Quality  control  at  FMSO  is  effected  by  the  organizational 
element  that  produces  the  product  and  by  a  small,  central¬ 
ized  staff.  Improved  systems  development  and  a  higher  level 
of  quality  control  are  the  goals  of  FMSO.  The  recommenda¬ 
tions  and  conclusions  offered  are  based  on  an  extensive 
literature  search  of  existing  material  on  software  quality 
assurance,  an  indepth  study  of  selected  industry  quality 
assurance  departments,  and  an  examination  of  the  current 
state  of  quality  control  procedures  at  FMSO.  These  recom¬ 
mendations,  if  implemented,  should  serve  to  improve  the 
quality  control  at  FMSO  and  assist  the  organization  in 
achieving  their  goals. 

r; 


TABLE  OF  CONTENTS 


I.  INTRODUCT ION  .  9 

II.  QUALITY  ASSURANCE  TECHNIQUES . 12 

A.  QUALITY  AND  QUALITY  ASSURANCE  .  13 

1.  Quality  Defined . 13 

2.  Quality  Assurance  Defined . 14 

3.  Traditiofial  Quality  Assurance  in  Industry  15 

B.  THE  RCLE  OF  QUALITY  ASSURANCE . 17 

1.  Why  the  Need  for  Quality  Assurance?  ...  17 

2.  Objectives  of  Quality  Assurance . 18 

3.  Costs  of  Quality  Assurance . 19 

C.  SOFTWARE  DEVELOPMENT  .  .  .  . 20 

1.  Software  Lifecycle  Phases . 20 

2.  Software  Properties . 23 

3.  Hardware  Characteristics  vs.  Software 

Characteristics . 25 

4.  software  Management . 2n 

5.  Standards  for  Software  Development  ....  30 

D.  SOFTWARE  QUALITY  ASSURANCE  METHODOLOGY  ....  32 

1.  Quality  Assurance  Plaining . 32 

2.  Staffing  and  Organization . 36 

3.  Reviews  and  Audits . 42 

4.  Testing  . . ..45 

5.  Software  Quality  Assurance  Tools  and 

Techniques . 47 

6.  Software  Documentation . 50 

7.  Configuration  Management . 51 

III.  A  SURVEY  OP  MILITARY  AND  CIVILIAN  SOFTWARE  QUALITY 

ASSURANCE  PROGRAMS  .  53 

A.  INTRODUCTION . 53 

B.  SOFTWARE  QUALITY  ASSURANCE  PLAN  STANDARDS  .  .  54 


5 


1.  Military  Specification  52779A . 5ft 

2.  IEEE  Standard  for  software  Quality 

Assurance  Plans . 55 

C.  PHASES  OF  THE  SOFTWARE  DEVELOPMENT  PROCESS  .  .  56 

D.  QUALITY  ASSURANCE  PROGRAMS . 59 

1.  TRW  Defense  Systems  Group  . . 59 

2.  Naval  Ocean  Systems  center . 76 

3.  General  Electric  Company  .  87 

E.  CONCLUSION . 9ft 

17.  THE  QUALITY  APPROACH  AT  PHSO . 95 

A.  STRUCTURE . 95 

E.  THE  QUALITY  PROCESS . .98 

C.  QUALITY  ASSURANCE  VS.  QUALITY  CONTROL  ....  99 

D.  SPECIFIC  QUALITY  CONTROL  RESPONSIBILITIES  .  .100 

E.  TESTIHG  TO  ENSURE  QUALITY . 101 

1.  Types  Of  Testing . .101 

F.  SYSTEM  RELEASE  PROCEDURES . 102 

G.  EVALUATING  THE  QUALITY  PROGRAM . 103 

V.  RECOMMENDATIONS . 108 

LIST  OF  REFERENCES . lift 

INITIAL  DISTRIBUTION  LIST . 118 


LIST  OP  TABLES 


Software  Lifecycle  Comparison . 

Hardware  and  Software  Product  Life  Cycles 


LIST  OF  FIGURES 


2.1  Typical  Quality  Assurance  Organizational 

Structure . 16 

2.2  Quality  Assurance  Project  Costs . 19 

2.3  Relative  Error  Correction  Costs  .  21 

2.4  Relationships  Between  Software  Quality  Factors  .  26 

2.5  Computational  Paths  . . 27 

2.6  Estimated  Growth  of  software  Costs . 29 

2.7  Quality  Assurance  Activities  .  36 

2.8  Typical  Software  Quality  Plan  Outline  .  37 

2.9  Sample  EDP  Department  Organization  .  39 

2.10  Informatics  Inc.  Organization  Chart . 40 

2.11  Traditional  Organizational  Structure . 41 

2.12  Typical  Functions  of  a  QA  Dept.  During  Testing  .  47 

2.13  Tools  Used  by  Software  Phase . 48 

3.1  Software  Development  Process . 58 

3.2  Software  Quality  Assurance  Process . 59 


3.3  Development  Process  for  Quality  Software  ....  60 

3.4  TRW  Corporate  Organizational  Structure . 61 

3.5  TRW  Project  Organizational  Structure . 6  3 


3.6  Test  change  Request  . . 71 

3.7  Design  Problem  Report . 73 

3.8  Software  Problem  Report . 75 

3.9  Temporary  Modification  notice . 77 

3.10  NOSC  Organizational  Structure . 78 

4.1  Organizational  Structure . 96 

4.2  Project  Flow  Model . 97 


8 


■■  v 


I.  IN  T50D0CTI0N 


The  computer  industry  has  gradually  evolved  over  the 
years  from  the  massive  hardware  systems  that  were  very 
expensive  to  build  and  maintain  to  *he  present  state  of  the 
art  where  a  tiry  one  quarter  inch  square  chip  has  aou 
computing  power,  is  enormously  cheaper  to  buy,  and  can  be 
maintained  virtually  100  percent  of  the  time.  Along  with 
the  hardware  change  we  have  seen  an  even  greater  improvement 
in  the  software  development  from  simple  mathematical  compu¬ 
tation  to  the  launching  and  recovery  of  manned  space 
vehicles  with  greater  reliability  and  performance  than  at 
any  other  time  in  history. 

This  evolution  has  not  happened  by  accident.  The  devel¬ 
opment  has  been  based  upon  making  mistakes,  documenting 
those  mistakes,  and  proceed ing  on.  As  people  continued  the 
process  they  studied  past  documentation  and  continued  the 
documentation  process  until  it  has  become  an  accepted  part 
of  the  development  cycle.  Although  no  systematic  approach 
was  utilized,  there  has  been  an  ever  increasing  tendency 
towards  the  development  of  a  set  of  standards  that  could 
provide  an  avenue  for  common  understanding  with  a  minimum  of 
confusion. 

The  growth  of  systems  software  which  occured  during  the 
past  two  decades  has  presented  each  organization  with 
continuing  challenges  in  maintaining  an  effective  organiza¬ 
tional  structure  and  in  following  efficient  systems 
development  methods  and  strategies  which  can  be  transferred 
from  one  entity  to  another  with  a  maximum  degree  of  cohe- 
siveness  and  precision  and  a  minimum  of  ambiguity  and 
con  fusion. 
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According  to  D.  Ross  of  SofTech  Inc.  [Ref.  1],  -.he 
qualify  of  software  is  relative  to  *:he  intended  application 
and  car.  only  be  achieved  by  a  disciplined  methodology  in 
which  quality  requirements  are  initially  applied  to  the 
original  requirements  definition  for  the  problem  and  are 
than  carefully  checked  and  confirmed  at  every  stage  in  the 
production  process.  In  order  to  have  any  sensible  treatment 
of  quality  every  aspect  of  the  system  life  cycle  must  be 
based  on  an  orderly,  controlled,  and  disciplined  metho¬ 
dology.  This  is  not  to  say  that  all  software  must  be 
produced  exactly  with  the  same  set  of  tools  and  techniques, 
for  this  clearly  would  be  excessive.  Thee*  say  be  a  broad 
spectrum  of  the  degree  to  which  the  ideal  system  technology 
is  approached,  tut  ever,  the  simplified,  streamlined  metho¬ 
dologies  must  be  complete  and  consistent.  Each  version  must 
be,  in  some  reasonable  fashion,  a  proper  degeneration  of  the 
elaborate,  most  advanced  state  of  the  art,  merely  simplified 
to  suit  a  simpler  set  of  circumstances.  If  must  be  incum¬ 
bent  upon  all  persons  that  are  engaged  in  the  project  to 
combine  company  standards  and  common  sense  to  arrive  at  the 
required  level  that  will  ensure  a  quality  product. 

To  accomplish  this,  there  exists  within  each  organiza¬ 
tion  discussed  in  this  paper  a  group  whose  primary 
responsibility  is  ensuring  that  company  standards  are 
enforced.  The  Quality  Control/  2uality  Assurance  Branches 
are  the  organiza  ‘ions  tasked  with  the  job.  It  is  therefore 
the  purpose  of  this  paper  to  examine  the  Qualify  Control 
programs  of  various  government  and  non-government  organiza¬ 
tions  that  produce  software  and  have  established,  well 
documented  standards.  The  effectiveness  of  their  program 
and  the  method  of  operation  within  their  company  structure 
will  also  be  discussed. 
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In  Chapter  2,  the  authors  will  list  and  identifv  currer- 
trends  and  state  of  the  art  processes,  techniques,  and 
methods  that  have  been  collated  through  an  exam ir.at icr.  of 
current  literature  about  quality  and  the  software  develop¬ 
ment  process.  In  Chapter  3  and  Chapter  4  we  will  discuss 
the  Quality  Control  programs  at  TPM,  General  Electric,  Naval 
Oceans  Systems  Command,  and  the  Fleet  Material  Support 
Office.  Finally,  in  Chapter  5,  a  set  of  recomendations  with 
justification  will  be  provided  for  FSSO  consideration  during 
the  planning  and  execution  of  future  corporate  policy  and 
expansion  with  emphasis  on  the  Quality  Control  effort. 
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ii.  fiojLiri  assurance  I ec uniques 


The  rapid  expansion  of  the  computer  industry  in  the  past 
ten  years  has  been  accompanied  by  an  increase  in  the  problem 
of  producing  a  quality  software  product.  Surveys  have  shown 
that  as  much  as  60  percent  of  software  produced  have  serious 
faults  in  the  first  iteration  -  faults  serious  enough  to 
cause  the  prograa  to  fail  in  its  task  [Ref.  2].  A  tradi¬ 
tional  facet  of  American  manufacturing  has  been  strict 
adherence  to  quality  standards  and  production  zi  soft¬ 

ware  should  be  no  different.  Although  computer  software  is 
sometimes  considered  more  of  an  art  than  an  engineered 
product,  computer  professionals  agree  that  software  quality 
assurance  is  an  important  part  of  the  software  life  cycle 
and  most  data  processing  departments  now  contain  seme  sort 
of  guality  assurance  function.  Many  cf  the  significant 
elements  of  production  engineering  such  as  documentation, 
testing,  project  management  and  quality  assurance  are  new 
emerging  as  an  integral  part  of  a  software  production 
activity.  Such  things  as  structured  programming,  software 
engineering  and  guality  assurance  are  fast  becoming  the  norm 
rather  than  the  exception.  User  satisfaction,  compliance 
with  approved  methods  of  building  applications,  organiza¬ 
tional  goals,  and  performance  goals  have  all  been  driving 
forces  behind  this  movement.  This  chapter  deals  with 
quality  assurance  in  the  computer  industry  and  its  role  in 
software  development. 
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A.  QUALITY  AHU  QUALITY  ASS  USANCE 


1 .  Quality  Eef  ir.ed 

Quality  is,  at  best,  a  relative  and  subjective 
measurement.  The  American  Heritage  Dictionary  defines 
quality  as  a  characteristic  or  attribute;  a  property.  It 
also  is  thought  of  as  the  natural  or  essential  character  of 
something.  In  everyday  life,  quality  measurements  are  done 
continually  and  usually  without  second  thought. 
Side-by-side  comparisons  of  objects  under  identical  condi¬ 
tions  and  with  predetermined  concepts  form  the  basis  of  most 
ccmparitive  judgements.  3  nf ortur.atsiy,  these  decisions  are 
usually  unique  and  have  little  value  to  anyone  else  unless 
they  are  made  by  an  expert  (Ref.  3].  One  widespread  opinion 
is  that,  by  its  very  nature,  quality  defies  definition  and 
must  be  uniquely  defined  for  the  item  in  question  by  stating 
a  list  cf  characteristics  and  attributes.  This  technique 
implies  no  evaluation  or  judgement  of  the  item,  but  merely 
provides  descriptive  traits  by  which  the  appraiser  may  form 
an  opinion  fRef.  h].  Ken  Johnson,  a  software  quality  assu¬ 
rance  manager  in  industry  and  chairman  of  a  working  party 
set  up  by  the  Electronic  Engineering  Association  concerned 
with  software  quality  assurance,  iisaqrees  somewha-  with 
other  definitions  concerning  the  ephemeral  attribute  of 
quality  and  declares  that  quality  is  "the  totality  of 
features  and  characteristics  of  a  product  or  service  which 
bear  on  its  ability  to  satisfy  a  given  need.  In  short,  it 
is  a  fitness  for  a  purpose  at  an  economic  cost"  [Ref-  2],  A 
recent  study  pointed  to  a  number  of  myths  concerning  quality 
in  organizations.  These  include  such  things  as  quality  is 
impossible  to  measure,  quality  lowers  productivity,  quality 
means  poor  workers,  and  quality  is  the  responsibility  of  the 
"quality  department".  It  goes  on  to  give  the  sharpest  defi¬ 
nition  of  qualit y-"quality  is  the  sum  costs  of  prevention. 
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This  definition 


appraisal  and  failure"  [Bef.  5].  This  definition  sheds 
lighr  on  the  subject  but  is  still  not  as  comprehensive  as 
may  be  desired.  Hence,  expectations  concerning  quality 
measurements  must  be  tempered  with  realistic  knowledge  that 
any  measurement  will  be  partially  imperfect  or  imprecise. 
When  dealing  with  software,  in  itself  not  the  most  tangible 
of  products,  confidence  levels  and  error  tolerances  play  an 
important  role  in  determining  acceptable  quality  levels. 

2.  Quality  Assurance  P  ef  insd 

Unlike  the  concept  of  quality,  which  is  usually 
Thought  cf  as  an  attribute  of  a  good  or  service,  quality 
assurance  is  most  often  related  to  a  process  cr  methodology. 
According  to  Frank  Ingressia  from  TRW  Corporation's  Defense 
and  Space  Systems  Group  (DSSG),  "quality  assurance  is  a  lot 
like  sex,  freedom  and  democracy.  Everyone  is  for  in,  but 
only  under  certain  conditions."  There  are  a  host  of  defini¬ 
tions  which  apply  to  quality  assurance.  The  official  Air 
Force  position  is  that  quality  assurance  is  a  discipline 
which  provides  adequate  assurance  that  material,  data, 
supplies  and  services  conform  to  established  technical 


requirements  and  achieve  satisfactory  results  [Hef.  6  ]- 
This  definition  is  much  more  encompassing  than  most  early 
interpretations  which  called  for  quality  assurance  to  merely 
verify  conformance  to  specifications.  At  the  other  end  of 
the  spectrum  is  the  feeling  that  qualify  assurance  is  merely 
the  business  of  ensuring  that  the  product  is  not  the  result 
of  good  luck  but  rather  the  inevitable  reward  for  good 
management  practices.  Still  another  definition  and  perhaps 
cne  that  will  become  widely  used  has  been  proposed  by  the 
Institue  of  Electrical  and  Electronic  Engineers  in  their 
recent  study  concerning  software  quality  assurance  standards 
(Hef.  7j.  It  states  that  "quality  assurance  is  a  planned 
and  systematic  pattern  of  all  actions  necessary  to  provide 


14 


adequate  confidence  that  the  item  or  product  cor. f errs  to 
established  technical  requirements".  This  definition  has 
been  borrowed  frea  MIL-STD-109B  and  is  consistent  with  the 
accepted  usage  of  the  tera.  Conforming  to  specifications, 
ensurinq  good  management  practices,  testing  of  requirements, 
confidence  that  the  system  is  reliable  or  the  product  is 
desirable — all  of  these  things  are  considered  goals  of  a 
quality  assurance  plan.  The  bottom  line  for  a  quality  assu¬ 
rance  function  is  ensuring  that  user's  needs  have  been 
adequately  satisfied.  There  are  a  multitude  of  different 
philosophies  concerning  the  achievement  of  these  goals  and 
they  will  be  briefly  addressed  in  a  later  section  of  this 
cha  pt  er . 

3.  Traaitio nal  Quality  Assurance  in  Industry 

For  manufactured  products,  quality  usually  means  a 
combination  of  quality  of  design  and  of  manufacture 
[Ref.  8].  Quality  of  design  is  the  value  inherent  in  the 
design;  a  measure  of  the  excellence  of  the  design  in  rela¬ 
tion  tc  the  customer's  requirements.  The  quality  controller 
has  the  responsibility  to  ensure  that  the  quality  level 
determined  by  management  can  be  achieved  on  production 
equipment.  Quality  of  manufacture  is  a  measure  of  how  well 
the  product,  at  acceptance,  conforms  to  the  design.  There 
are  most  often  five  basic  stages  of  quality  control  in  a 
factory  [Ref.  8].  These  consist  of;  (1)  Deciding  what  to 
manufacture  and  prepare  specifications  covering  all  require¬ 
ments,  (2)  Make  pre-pr  oducticn  checks  and  work  out 
organizational  responsibilities,  (3)  Production,  (4) 
Feedback  on  quality  deficiencies,  and  (5)  Establish  long¬ 
term  quality  plans. 

The  quality  control  function  in  a  manufacturing 
plant  is  usually  a  very  complex  and  intricate  organization. 
It  becomes  involved  in  all  phases  of  the  production  process 
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and  serves  as  the  ongoing  checker  for  production  results. 
The  role,  structure  and  objectives  of  quality  assurance  in 
software  production  will  be  examined  in  detail  later  in  the 
chapter,  but  there  are  many  siailarities  which  exist  and 
indeed  industrial  quality  assurance  forms  the  basis  for 


Pigure  2.1  Typical  Quality  lssarar.ce  Organizational  Structure. 

software  quality  assurance  techniques.  Figure  2.1  illus¬ 
trates  a  typical  manufacturing  quality  assurance  department. 


B.  THE  HOLE  OF  QUkLITY  ASSURANCE 


1  •  W hy  the  Need  foe  3u ality  Assurance? 

The  statistics  concerning  the  growth  of  the  software 
industry  in  the  past  15  years  as  well  as  the  problems 
concerning  this  growth  are  quite  well  documented  and  recog¬ 
nized.  For  example,  one  company,  Boeing  Aerospace,  reported 
that  on  a  large  software  project:  a),  only  14  percent  of  the 
total  number  of  runs  would  have  been  required  had  there  been 
no  errors  or  failures,  and  b) .  39  percent  of  the  runs,  while 
successfully  completed,  were  later  invalidated  because  of 
data  errors,  tape  failures,  or  program  bugs  [Ref.  9].  From 
a  cost  standpoint,  over  eight  billion  dollars  was  spent  for 
software  of  various  types  in  1980  [Raf.  10].  Computers,  and 
in  turn  software,  are  becoming  entangled  in  every  aspect  of 
our  daily  lives.  From  electronic  banking  and  shopping  to 
NASA  space  projects  to  mora  effecient  use  of  farm  machinery, 
we  rely  on  computer  software  to  help  us  make  more  and  more 
of  our  decisions.  Software  developers  have  recognized  their 
responsibility  towards  guality  software  and  most  data 
processing  departments  now  incorporate  some  sort  of  quality 
assurance  function  into  the  production  of  software.  This 
quality  assurance  function  is  able  to  alleviate  the  problem 
most  software  managers  have  of  becoming  involved  in  the 
system  at  a  point  when  the  cost  becomes  significant  and  the 
dates  of  implementation  approach.  The  establishment  of  a 
quality  assurance  function  provides  management  with  a  degree 
of  confidence  that  an  independent,  technically  trained  group 
is  monitoring  the  goals,  methods  and  performance  of  applica¬ 
tions  from  the  beginning  of  the  project. 


2.  Oblectiv  eg  of  Quail  ty  Assurance 

The  quality  assurance  function,  as  part  of  the 
systems  production  group,  works  to  ensure  that  standards 
concerning  goals,  methods  and  objectives  are  met.  The 
quality  assurance  group  typically  perforins  those  functions 
that  the  data  processing  manager  might  do  personally  if  time 
permitted.  Quality  assurance  reviews  each  system  to  ensure 
compliance  wiht  the  following  items.  The  system  must  meet 
the  needs  of  the  user  department  and  other  users  and  at  the 
same  time  not  infringe  on  the  rights  of  other  systems  users. 
The  goals  of  the  system  should  be  consistent  with  the  objec¬ 
tives  of  the  entire  organization.  If  there  is  a  conflict, 
the  goals  of  the  organization  should  maintain  priority  ever 
the  goals  of  one  user.  The  system  goals  should  also  mesh 
with  the  2DP  department  objectives  and  if  there  is  a 
conflict,  it  should  be  resolved  before  implementation.  If 
there  are  external  industry  or  government  requirements,  the 
goals  of  the  system  should  conform  to  these  standards. 
Controls  on  the  system  must  be  complete  (management 
controls)  and  the  system  must  be  auditable.  The  system 
should  conform  to  all  general  policies,  procedures,  stan¬ 
dards  and  guidelines  established  by  the  organization  and  the 
electronic  data  processing  department.  Quality  assurance 
must  finally  ensure  that  the  design  of  the  system  is  econom¬ 
ical  (least  cost  system),  effective  (desired  results  with 
minimum  effort),  and  efficient  (maximize  use  of  people  and 
machines) . 

3.  Costs  of  Quality  As surance 

The  cost  of  a  quality  assurance  function  is  very 
difficult  to  estimate  or  even  measure.  William  Perry,  an 
author  of  extensive  material  on  software  quality  assurance 
and  a  member  of  the  Quality  Assurance  Institute  in  Orlando, 


Florida,  comes  closest  to  a  precise  figure  by  saying  '•hat 
"if  quality  assurance  is  included  as  a  line  item  rn  a 
project's  budget,  it  should  range  somewhere  between  2.5  and 
5  percent  of  the  total  project  cost"  [Ref.  4].  Figure  2.2 
is  indicative  of  the  percentage  of  costs  expended  on  a 
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Figure  2. 2  Quality  Assurance  Project  costs. 

typical  five-phase  software  development  project.  This  esti¬ 
mate  is  still  rather  idealistic  because  of  the  differences 
which  may  arise  because  of  staffing  alternatives,  metho¬ 
dology,  lifecycle  entry,  the  difficulty  in  defining  quality 
assurance  and  other  unknowns. 

Cost  justification  is  an  important  aspect  of  imple¬ 
menting  a  quality  assurance  function.  In  the  hardware 
arena,  the  cost  cf  quality  assurance  is  most  often  justified 
by  lower  warranty  cost  markups  in  the  price  of  the  product. 
This  is  equally  true  in  the  software  world.  Warranty  costs 
will  be  lower  for  a  quality  software  product.  Another 
fringe  benefit  of  a  quality  software  product  is  its  ease  of 
adaptability  to  a  similar  product  at  a  lower  cost  [Ref.  11]. 


Ths  cost  savings  of  an  efficient  quality  assurance  activity 
is  often  hard  to  justify  because  of  the  indiscipline  of 
those  not  quality-conscious  often  makes  measurement  of 
improvements  very  difficult.  As  is  usually  the  case, 
results  speak  the  loudest.  The  cost  savings  involved  in 
having  projects  done  on  time  and  within  budget  allows 
quality  assurance  tc  maintain  its  lave!  of  efficiency  and 
reduces  management  time  spent  sorting  out  the  mess  which 
results  from  bad  planning. 

Finally,  it  has  been  shown  that  the  entry  point  of 
the  quality  assurance  function  into  the  lifecycle  of  the 
project  has  a  definite  effect  on  the  cost  [Bef.  12].  The 
scope  and  structure  of  the  quality  assurance  effort  is 
affected  strongly  by  the  cost  of  errors  related  to  the  phase 
of  development.  Figure  2.3  is  a  typical  illustration  of  the 
economics  of  error  detection  in  the  various  phases  of  devel¬ 
opment.  As  this  figure  shows,  the  earlier  a  problem  is 
detected,  the  less  expensive  is  the  cost  of  correction. 

C.  SOFTWARE  DEVELOPMENT 

1 .  Software  Lifecycle  Phases 

Referring  to  the  lifecycle  of  software  is  the  most 
common  method  of  addressing  the  development  of  a  software 
product.  A  review  of  the  liturature  has  produced  a  plethora 
of  illustrations  of  what  is  considered  to  be  the  true  or 
ideal  lifecycle.  Most  examples  use  different  terminology  to 
describe  what  is  happening  at  a  particular  stage  in  the 
lifecycle,  but  they  all  tend  to  incLude  the  critical  items. 
The  simplest  lifecycle  found  is  one  in  which  there  are  only 
three  stages  -  design  and  development,  active  and  passive 
[Ref.  13].  Conversely,  the  most  complex  definition  of  a 
software  lifecycle  contains  eight  phases-systems  definition, 
software  allocation,  specification,  design,  code. 
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Figure  2. 3  Relative  Error  Correction  Costs. 

verification,  integration,  and  operation  [Ref.  14].  Table  1 
depicts  a  comparison  of  the  various  phases  of  the  lifecycle 
of  software. 

whatever  the  phases  of  software  development  are 
called,  there  are  certain  items  which  must  be  accomplished. 
The  beginning  of  a  project  may  be  designated  as  a  feasi¬ 
bility  study,  requirements  definition,  systems  definition, 
user  requirements  study,  initiation  study  cr  something  else, 
but  it  consists  of  all  activities  which  deal  with  deter¬ 
mining  whether  or  not  a  software  project  should  be 
initiated.  Such  things  as  cost-benefit  studies,  goal  defi¬ 
nitions,  and  documentation  requirements  are  typical 
activities  which  should  be  accomplished  here.  Next  comes 
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Software  Lifecycle  Comparisor. 


AUTHOR  LIFECYCLE  PHASES 

PERRY  - - -  Feasibility  -  Design  -  Programming  -  Testing  - 

Conversion 


FUJII  - 

MENDIS 

ROBERTS 

HOW  LEY 


Conceptual  Design  -  Requirements  Definition  - 
Design  -  Code  and  Checkout  -  Testing  - 
Integration  -  Operational 

Design  -  Code  and  Debug  -  Qualification  Test 

Design  and  Development  -  Active  Stage  - 
Passive  Stage 

Systems  Definition  -  Software  Allocation  - 
Verif ication ‘ -  Integration  -  Operation 


DONN  ana 

ULLMAN - User  Reouirem  er.ts  -  svstem  Functional  Specs  - 

Software  Functional  Specs  - 
Implementation  -  Verification  and  Test  - 
Operations  and  Maintenance  - 
Configuration  Management 

FIPS  PUB  38  -  Initiation  -  Development  -  Operation 


the  general  design  of  the  system.  This  stage  is  normally 
labeled  design,  design  and  development,  software  design,  or 
systems  functional  specifications.  Activities  such  as 
design  alternatives,  specific  requirements,  functions  to  be 
performed,  and  program  and  data  base  specifications  should 
be  included  in  this  phase.  The  next  phase  is  probably  the 
most  rudimentary  in  terms  of  work  and  deals  with  programming 
and  testing.  This  stage  is  also  referred  to  as  coding  and 
debugging,  verification  and  validation  (after  codrng  is 
.complete),  or  is  sometimes  broken  into  two  distinct  phases. 
The  system  is  now  written  in  the  desired  language  and 
various  tests  are  performed  to  ensure  the  system  performs  as 
desired.  The  next,  and  most  often  Last  phase,  is  referred 
to  as  conversion,  integration,  operations,  implementation, 
maintenance,  or  configuration  maaagement.  This  stage 
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consists  of  maintaining  the  software,  perfonir.g  ongoing 
evaluations  and  changing  in  as  additional  requirements  are 
identified. 

2.  Software  properties 

As  stated  previously,  software  is  an  elusive  product 
upon  which  to  place  a  quality  measurement.  Howley  and  Fink, 
software  quality  engineers  for  the  Boeing  Aerospace 
Corporation,  have  attempted  to  verbalize  what  a  quality 
software  product  is  by  stating  "A  quality  software  product 
may  be  defined  as  one  whioh  exhibits  the  following  proper¬ 
ries:  it  satisfies  the  software  specification  and  design 

requirements;  it  performs  all  intended  functions;  it  is 
relatively  free  of  design,  interface  and  coding  deficien¬ 
cies;  it  has  a  low  life-cycle  cost;  it  is  properly 
identified  and  documented;  and  it  incorporates  all  needed 
software  quality  characteristics"  [Ref.  15].  To  achieve  the 
level  of  quality  which  is  desired  by  the  above  definition, 
there  are  a  number  of  factors  which  contribute  to  che 
production  of  quality  software  [Ref.  16].  These  include, 
but  are  not  limited  to: 

1.  Correctness  -  This  generally  leans  programs  perform 
in  exactly  the  manner  specified  in  the  program  docu¬ 
mentation.  Correctness  is  usually  considered  an 
ideal  quality  which  is  rarely  achievable. 

2.  Reliability  -  This  attribute  means  that  programs 
perform  relatively  trouble  free  all  the  functions 
expected  from  the  specifications  or  documentation. 

3.  Validity  -  Validity  is  concerned  with  the  question  of 
whether  the  functions  and  performance  of  the  programs 
are  adequate  and  suitable  to  a  needed  purpose.  The 
software,  without  manual  intervention  or  additional 
programming,  should  perform  the  functions  that 
reasonably  would  be  expected  of  it.  This  attribute 
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is  a  very  subjective  or.e  an}  must  be  flexible  to 
changing  requirement s. 

а.  Resilience  -  This  means  -.hat  programs  should  be 
designed  in  such  a  way  to  be  forgiving  of  common  user 
and  data  errors.  Inconsistent  or  unacceptable  data 
entries  shouldn't  provoke  actions  which  make  no  sense 
to  the  user. 

5.  Usability  -  Human  factors  and  limitations  and  conve¬ 
nient  usage  techniques  should  be  considered  whenever 
a  program  is  written. 

б.  Clarity  -  Programs  should  be  easily  understandable 
from  the  users  manual  ana  all  the  iocumsr.-.a". ion 
should  be  clear,  concise  and  cogent.  Programs  should 
be  modularly  designed,  have  explanatory  comments 
where  necessary,  and  use  meaningful  choices  of  vari¬ 
able  names. 

7.  Maintainability  -  3aod  documentation  and  comments  as 
well  as  clear  structure  will  make  programs  more 
easily  repairable.  Clarity  is  also  essential  for 
making  minor  improvements. 

8.  Modifiability  -  Major  changes  should  be  anticipated 
and  the  software  designed  so  that  program  functions 
that  might  require  major  change  are  well  documented 
and  isolated  in  distinct  modules. 

9.  Generality  -  Programs  should  be  applicable  to  a  wide 
range  of  input  values  and  usage  modes. 

10.  Portability  -  Programs  should  be  easily  adaptable  to 
transfer  to  another  computer  system  or  operating 
system. 

11.  Testability  -  Programs  should  be  simply  structured 
and  use  geiieraly  algorithms,  to  facilitate  step-by- 
step  testing  of  all  capabilities. 

12.  2fficiency  -  The  attempt  should  be  made  to  keep  the 
cost  of  program  operation  as  low  as  possible. 
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The  preceding  list  contains  many  cf  the  attributes 
of  a  quality  software  product  but  is  not  the  only  list  avai¬ 
lable.  Other  authors  include  such  things  as  integrity, 
flexibility,  reusability,  interoperability,  and  others  which 
are  descriptors  cf  quality  software.  Figure  2.4  is  a  good 
illustration  of  how  these  factors  affect  each  other  and  what 
degree  of  a  certain  factor  is  required  when  a  different 
factor  is  recognized.  As  can  be  seen,  some  factors  are 
synergistic  while  others  conflict.  The  impact  of 
conflicting  factors  is  that  the  cost  tc  implement  will 
increase.  This  will  serve  to  lower  benefit  to  cost  ratios 
[  Bef  .  17]. 

3.  Har dwar  e  Char  a vs.  Software 

Characteristics 

Hardware  is  a  tangible  piece  of  engineering.  It  has 
very  precise  sp ecif icat ion s  and  drawings  and  is  based  on 
well  established  building  principles,  the  aim  being  to  manu¬ 
facture  many  identical  (or  near  identical)  items.  The 
design-development- production  cycle  is.  mature  ar.d  well 
tuned.  Refinements  to  the  design  may  be  made  many  times 
before  a  commit  rent  to  manufacture  is  made.  In  contrast, 
software  engineers  ship  their  prototypes.  software  is  a 
largely  intangible  product,  only  described  by  many  volumes 
of  specifications  and  listings.  Software  is  unlikely  to  go 
through  as  many  prototype  stages  and,  therefore,  the  oppor¬ 
tunity  for  design  iteration  and  improvement  is  somewhat 
limited  [Ref.  18]. 

Most  aspects  of  hardware  are  functionally  testable 
and  have  very  specific  requ ireaents  testing  programs.  It  is 
fenced-in  by  established  principles  and  well-known,  widely- 
used  disciplines.  Unlike  hardware,  software  is  functionally 
non-  testable  in  all  but  the  simplest  of  computer  programs 
and  as  a  result,  it  is  very  difficult  to  test  software 
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completely  [Ref.  19].  Figure  2.5  illustrates  *he  problem  of 
testing  software.  Each  circle  represents  a  processing  node. 
The  clockwise  arcs  are  jumps  around  the  individual  nodes, 
and  the  counter-clockwise  arcs  specify  the  number  of  itera¬ 
tions  of  each  half.  The  number  of  discrete  states  possible 
within  this  trivial  diagram  is  approximately  100  quadrillion. 

If  these  could  be  tested  at  the  staggering  rate  of  one 
per  microsecond,  it  would  have  been  necessary  to  start  the 
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Pigare  2.5  Compa tational  Paths. 

testing  over  2,  0  00  year-  ago  to  **»*  r. ex*  north’s  scheduled 
delivery. 

The  usual  cause  of  hardware  failure  is  component 
deterioration.  software  failures  are  almost  always  design 
errors  that  show  up  only  when  the  software  is  used  under 
certain  conditions.  Hence,  Quality  Assurance  techniques  for 
software  focus  on  getting  the  design  right  [Ref.  20]. 

A  comparison  of  hardware  and  software  lifecycles  is 
offered  by  Table  2  and  shows  clearly  that  similar  terms  in 
the  twc  fields  sometimes  have  radically  different  meanings. 

4 .  Software  Ma naqement 

The  driving  forces  behind  implementing  management 
structure  in  any  organization  are  reduction  of  costs, 
increased  control,  and  production  of  a  quality  product. 
Software  production  is  no  different.  Figure  2.6  depicts  the 
rise  in  software  costs  in  *he  last  twenty  years.  As  is 
apparent,  software  costs  greatly  exceed  equipment  costs  over 
the  useful  life  of  computer  services.  With  this  kind  of 
growth  it  is  imperative  that  software  be  managed  to  minimize 
costs.  While  cost  minimization  is  an  important  aspect  of 
software  management,  there  are  other  reasons  of  equal  impor¬ 
tance.  Most  software  in  production  today  is  a  complex 


27 


TABLE  II 

Hardware  and  Software  Product  Life  Cycles 


HARDWARE 


SOFTWARE 


DEVELOPMENT 


Determine  user 
Requirement  s 
Develop  Product 

Concept  (Fun  ctional) 
Specify  Component 
Design  (Detailed) 
Build  and  Test 
Prototype 

Develop  Manufacturing 
Techniques 


Determine  user 
Requirements 
Develop  Product 

Concept  (Functional) 
Specify  Component 
Design  (Detailed) 
Implement  and  Test 
Programs 


I  Manufacture  Product 
INSTALLATION!  Make  Product  Available 
|  to  Users 


Copv  Programs 
Make  Program 

Availaole  to  Users 


Maintenance  (Correct 
Component  Failures) 
Recall  Product  to 

MAINTENANCE-  Correct  Des ian  Flaws 
IMPROVEMENT 

Enhance  Product 


Maintenance  (Correct 
Implementation  and 
Design  Errors) 
Maintenance (Adapted 
to  Chanced  User 
Environment) 


PHASE-OUT 


Unit  is  Unusable  and 
Unrepairabi  e  (Replace) 

Product  is  Obsolete  Product  is  Obsolete 


technical  activity  that  must  be  directed  effectively.  The 
complexity  cf  any  program  in  all  but  the  simplest  of  appli¬ 
cations  is  such  that  programming  has  evolved  past  a  routine 
effort  that  can  go  unsupervised  or  be  done  by  junior 
personnel.  Any  investment  of  funds  or  resources  is  likely 
to  be  a  major  one  for  any  organization  and  the  technical 
choices  may  have  widespread  effects  throughout  the  organiza¬ 
tion.  Management  and  technical  control  by  professionals  is 
essential  for  resolving  design  issues  and  giving  adequate 
direction  to  programmers  regarding  cost  and  schedule 
targets.  Software  management  also  provides  for 


Figure  2.6  Estimated  Growth  of  Software  Costs. 

accountability  of  project  decisions  and  objectives  ar.l  gives 
top  management  visible  measures  of  success  in  the  accom¬ 
plishment  of  goals.  As  a  general  rule,  software  projects 
are  often  initiated  by  management  personnel  who  control 
budgets  and  schedules  and  software  designers  frequently  end 
up  doing  a  substantial  amount  of  independent  work  with 
little  pressure  to  evaluate  their  progress  or  the  remaining 
work  or  costs.  With  a  software  managment  structure  in 
place,  important  issues  and  objectives  such  as  cost,  quality 
and  schedule  can  be  carefully  evaluated  and  the  appropriate 
responsibility  for  the  decisions  assigned.  Technical 
controls,  working  procedures  ar.d  resource  management  are 
further  justifications  of  a  strong  software  management 
structure.  Questions  about  system  feasibility,  system 
quality,  design  methodology  and  testing  procedures  are  ones 
which  should  be  answered  by  software  management.  All  of 
these  things  help  designers  and  programmers  to  organize  and 
direct  their  efforts  efficiently  in  solving  such  problems 
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within  available  cost  and  t  ims  limits.  Another  raascr.  for 
software  management  is  the  ongoing  problem  of  maintenance 
after  implementation.  Some  estimates  put  the  cost  of 
maintenance  at  70*  of  total  software  costs.  It  is  important 
to  recognize  that  maintenance  is  just  as  important  as  devel¬ 
opment.  Good  software  management  principles  will  attempt  to 
be  ongoing  throughout  the  lifecycle  and  will  make  the 
strongest  effort  to  stress  close  management  of  affort  toward 
tha  most  needed  software  capabilities. 

The  management  of  software  development  is  often 
referred  to  as  "software  engineering".  This  implies  -hat 
the  principles  cf  production  engineering  management  can  be 
transferred  or  applied  to  software  development.  Software 
engineering  suggests  that  "the  entire  development  of  a  soft¬ 
ware  product  from  initial  conception  through  design, 
implementation,  testing,  and  maintenance  can  be  organized  in 
a  systematic  and  manageabla  fashion.  It  should,  therefore, 
be  possible  to  monitor  the  quality,  performance  and  cost  cf 
the  end  product  through  the  several  phases  of  its  life 
cycle"  [Ref.  21]. 

5.  Standard s  for  Software  Development 

An  area  which  has  been  seriously  neglected  in  the 
software  development  industry  during  its  growth  has  beer,  the 
establishment  of  standards  cf  conformance.  There  has  been  a 
recognition  of  this  lack  during  the  past  few  years  and 
attempts  have  been  made  to  provide  adequate  standards.  The 
most  widely  used  military  document  concerning  standardiza¬ 
tion  of  quality  assurance  plans  is  MIL-S-52779A  dated  1 
August  1979.  This  document  is  applicable  to  Department  of 
Defense  agencies  when  acquiring  software  where  the  acquisi¬ 
tion  involves  either  software  alone  or  software  as  a  portion 
of  a  system  or  subsystem.  It  provides  specific  guidance 
concerning  software  quality  assurance  program  requirements 
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and  covers  such  things  as  tools,  techniques  and  methodolo¬ 
gies,  computer  program  design,  wori c  certification,  cocuaea- 
tation,  computer  program  library  controls,  reviews  and 
audits,  configuration  management  and  testing  [Ref.  22]. 

This  document  is  used  not  only  by  DOD,  but  has  also 
been  referred  to  by  many  civilian  organizations  in  the 
absence  of  anything  better.  The  Institute  of  Electrical  and 
Electronic  Engineers  has  recently  sponsored  a  committee  to 
evaluate  the  problem  and  develop  a  set  of  software  quality 
assurance  standards.  Their  stated  purpose  was  to  "provide 
uniform,  minimum  acceptable  requirements  for  the  preparation 
and  content  of  software  quality  assurance  plans"  [Ref.  23], 
The  sections  of  the  standard  developed  contains  direction 
concerning  such  ■‘■hir.gs  as  reference  documents,  management, 
documentation,  standards,  practices,  and  conventions, 
reviews  and  audits,  con  figuration  management,  problem 
reporting  and  corrective  action,  tools,  techniques,  and 
met hcdolcgies ,  code  control,  media  control,  and  supplier 
control.  As  can  be  seen,  it  is  extensive  and  comprehensive 
in  scope  and  provides  guidance  for  development  of  a  thorough 
software  quality  assurance  plan. 

The  preceding  two  documents  are  the  most  widely  used 
standards  referred  to  when  developing  a  software  quality 
assurance  plan.  There  are  some  other  publications  which  can 
be  reviewed  for  direction  concerning  development  of  soft¬ 
ware.  Federal  Information  Processing  standards  Publication 
38  (FIPS  PUB  38)  provides  thorough  and  comprehensive  guide¬ 
lines  for  documentation  of  computer  program  and  automated 
data  systems.  National  Bureau  of  Standards  Special 
Publication  500-11  is  a  good  overall  guideline  to  computer 
software  management  and  quality  control.  There  are  a  host 
of  journal  and  proceedings  articles  dealing  with  software 
quality  assurance  which  provide  information.  All  of  these 
are  not  "official"  guidelines  and  lack  any  authoritative 
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endorsement.  The  best  hops  for  better  results  from  sziz 
quality  assurance  programs  lies  in  an  acceptance  ;;  1. : 

Standards  and  conformance  to  their  requirements. 
Unfortunately,  these  are  relatively  new  and  there  is  insuf¬ 
ficient  data  to  declare  that  new  industry  standards  have 
been  developed. 

0.  SOFTWARE  QUALITY  ASSURANCE  HETHODOLOGY 

1.  Quality  Assurance  P laming 

Planning  is  essential  for  the  successful  achievement 
•of  ar.y  project  ar.i  must  renair.  dynamic  to  be  of  any  use.  I*- 
is  important  that  plans  are  modified  tc  reflect  changes  in 
requirements  as  they  occur.  On  a  bread  scope,  a  quality 
assurance  plan  must  indicate  -he  particular  activities  which 
will  enable  the  required  level  of  quality  to  be  achieved  on 
any  giver.  project.  How  the  product  is  to  be  assured  and 
what  activi*ies  the  quality  assurance  group  is  zo  undertake 
in  order  to  satisfy  organizational  requirements  are  key 
elements. 

Quality  assurance  generally  parallels  the  systems 
development  process.  The  position  of  review  points  will 
depend  upon  management's  requirements  concerning  decision 
points  or  information  requirements.  The  importance  of  the 
system  to  the  organization  as  a  whole  will  determine  the 
amount  of  time  spent  on  each  project.  The  critical  points 
are  the  end  of  each  systems  development  life  cycle  phase. 
At  this  time,  an  opinion  is  rendered  by  quality  assurance  as 
to  the  adequacy  of  the  design  process  up  to  that  point. 
This  opinion  can  then  be  J3ed  as  a  decision  making  factor  in 
determining  whether  to  progress  to  the  next  phase.  In 
discussinq  the  review  criteria  a  five  phase  systems 
lifecycle  will  be  assumed. 
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The  feasibility  study  commonly  consists  of  evalua¬ 
tion  of  alternatives  and  techniques  to  solve  a  particular 
problem  and  recommend  a  course  of  action  to  management. 
This  may  or  may  not  include  a  computer  system  ar.d  the 
personnel  involved  in  the  study  may  or  may  not  have  computer 
experience.  The  role  of  quality  assurance  during  the  feasi¬ 
bility  study  is  usually  one  of  a  consultant  to  discuss  the 
practicality  of  alternatives  or  cost  estimates.  At  the  end 
of  the  feasibility  study,  quality  assurance  should  evaluate 
whether  the  study  team  followed  the  organization’s  proce¬ 
dures  in  developing  a  proposal  for  management  and  comment  on 
any  computerization  aspects  of  the  proposal. 

The  next  phase,  design,  is  critical  to  quality  assu¬ 
rance.  The  greatest  impac*  is  made  during  this  phase  ar.i 
the  quality  assurance  group  should  strive  to  impact  the 
design  without  actually  participating  in  the  design  process. 
The  goal  is  to  not  argue  for  or  against  particular  designs 
but  to  review  the  proposed  design  on  merit.  One  method 
which  is  widely  used  in  this  phase  Is  to  divide  the  design 
into  two  phases -inf ormal  and  formal.  The  informal  phase 
occurs  after  a  preliminary  design  is  on  paper  and  consists 
of  quality  assurance  giving  discussion  only  review  to  allow 
the  design  group  to  determine  if  they  are  on  the  right 
track.  There  is  no  report  to  management  generated  from  this 
phase.  A  structure  such  as  this  requires  a  good  working 
relationship  between  quality  assurance  and  the  design  group. 
At  ■'•he  end  of  the  design  phase,  a  formal  review  occurs  and 
the  design  is  normally  fixed  at  this  point.  Compliance  to 
performance  criteria,  systems  goals  and  procedures  are 
reviewed  by  quality  assurance. 

Program  design  and  program  coding  make  up  the  next 
phase  which  must  be  evaluated  by  quality  assurance.  While 
these  two  activities  may  be  combined  into  one  phase,  it  is 
usually  more  effective  and  it  facillitates  structured 
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programming  to  separate  them.  3y  reviewing  the  program 
design,  quality  assurance  has  a  greater  opportunity  to 
ensure  compliance  to  design  procedures  and  standards.  This 
will  hopefully  alleviate  many  problems  usually  encountered 
in  the  coding  phase.  At  the  end  of  the  programming  phase, 
quality  assurance  should  perform  a  review  to  ensure  compli¬ 
ance  to  procedures  and  standards  for  such  things  as  coding 
and  use  of  operating  system  facilities.  This  should  be  a 
detailed  review  and  qualify  assurance  must  examine  all 
aspects  of  coding,  operating  system  instructions,  file 
structures  and  anything  else  which  will  affect  the  operation 
of  the  computer. 

In  the  next  phase,  system  testing,  quality  assurance 
is  mostly  concerned  that  an  adequate  test  plan  has  bear, 
prepared,  that  it  is  followed  and  that  it  conforms  to  the 
standards  of  the  organization.  Quality  assurance  will  onlv 
review  rest  results  to  ensure  compliance  to  standards  and 
should  not  become  involved  in  a  detailed  test  plan.  kz  the 
end  of  the  system  testing  quality  assurance  should  again 
review  for  conformance  to  organizational  policy  and  chec1? 
for  user  satisfaction. 

The  last  phase  in  our  example,  implementation,  can 
be  the  broadest  in  scope  and  longest  in  duration.  This 
phase  is  sometimes  called  conversion  and  is  generally 
thought  of  as  the  process  of  replacement  cr  new  installment. 
As  with  testing,  the  primary  concern  of  quality  assurance  is 
that  a  bonafide  plan  has  been  defined  and  that  it  is  being 
followed.  The  completion  of  the  implementation  phase  brings 
the  final  review  by  quality  assurance  that  the  procedures 
defined  in  the  design  stage  were  followed.  Once  again,  user 
satisfaction  is  of  paramount  importance  and  quality  assu¬ 
rance  is  reviewing  plans  and  procedures. 
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The  foregoing  example  of  a  quality  assurance 
over  the  lifecycle  of  a  software  product  is  by  no  means  the 
only  one  to  be  used.  Another  typical  example  is  discussed 
by  Marilyn  Fujii,  a  software  quality  assurance  professional 
from  Logicon,  Inc. ,  in  which  the  lifecycle  is  divided  into 
seven  parts  and  quality  assurance  again  has  a  role  over  the 
entire  lifecycle  [Hef.  24],  The  early  stages  of  the  plan 
are  centered  arcund  defining  the  procedures  and  standards 
which  will  be  applicable  to  supporting  configuration  manage¬ 
ment  and  computer  program  development.  Most  other 
activities  consist  of  reviewing  and  auditing  software 
products  against  previously  set  standards.  Quality  assu¬ 
rance  is  responsible  for  all  design  reviews  and  audits  and 
they  evaluate  all  documentation  such  as  test  plan,  specifi¬ 
cations,  and  users  manuals.  Any  walkthroughs  or  acceptances 
testing  will  be  scheduled  and  conducted  by  guality  assu¬ 
rance.  At  the  delivery  point  in  the  lifecycle,  they  audit 
the  final  configuration  to  be  installed  in  the  operational 
environment.  Figure  2.7  offers  a  visual  presentation  of 
quality  assurance's  .role  in  this  example. 

The  examples  given  are  but  two  of  a  multitude  which 
can  be  found  in  professional  literature.  Regardless  of  what 
specific  method  is  used,  there  are  a  number  of  components 
which  should  be  included  in  any  software  quality  assurance 
plan.  The  plan  should  identify  procedures  to  be  used  in 
issuing  work  tasking  instructions  for  all  work  relating  to 
software  development.  Monitoring  of  procedures  and  assurina 
adherence  to  them  should  be  part  of  any  plan. 
Identification  of  schedules  and  resources  and  tracking 
progress  toward  them  should  be  included.  Work  descriptions, 
responsibility  assignments,  initiation  procedures,  report 
generation  procedures,  and  scheduled  completion  dates  should 
be  addressed.  The  plan  should  document  quality  assurance 
involvement  in  the  specified  development  program.  Also 
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Figure  2.7  Quality  Assurance  Activities. 

provided  should  be  visible  schedules,  milestones  and 
interdepartmental  dependencies  and  commitments.  Levels  of 
detail  required  should  be  included  in  any  plan.  Finally, 
the  organization  responsible  for  software  quality  assurance 
should  prepare  the  plan.  \  general  outline  of  a  typical 
quality  assurance  plan  is  provided  as  Figure  2.8, 

2.  Staffing  and  Organize  tl  on 

The  success  of  any  quality  assurance  function  begins 
with  the  personnel  assigned  to  the  staff.  Individuals  as 
knowledgable  as  senior  systems  analysts  and  designers  should 
ideally  be  assigned  to  quality  assurance.  It  is  not  enough, 
however,  for  quality  assurance  personnel  to  merely  possess 
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Figure  2.8  Typical  Software  Quality  Plan  Outline. 

the  characteristics  of  a  good  systems  analyst.  Quality 
assurance  personnel  must  command  the  respect  of  both  the 
individuals  whose  systems  are  being  evaluated  and  the 
management  of  the  EDP  department,  who  must  rely  on  them. 
The  ability  to  review  the  work  of  others  and  to  convince 
them  there  are  tetter  methods  to  perform  their  work  takes 
some  unique  skills  in  dealing  with  people.  Quality  assu¬ 
rance  reviewers  must  have  a  talent  for  good  communicative 
and  persuasive  ability  as  as  well  as  be  respected  for  their 
technical  ability. 
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There  are  a  number  of  variables  concerning  -he 
experience  and  number  of  people  assigned  no  gua_i-./  assu¬ 
rance.  The  size  of  rhe  project  an  hand  is  a  n-  -vor 
determining  factor.  Small  projects  which  appear  no  be-  rela¬ 
tively  simple  and  perhaps  repetitive  of  previous  -jobs  may  be 
short-changed  in  the  quality  assurance  area.  If  the  company 
itself  is  small,  it  may  not  be  able  to  afford  the  commitment 
to  quality  assurance  of  a  large  corporation.  Top  management 
may  not  recognize  the  need  for  qualify  assurance  and  hence 
give  if  less  than  prominent  attention. 

The  organization  of  a  quality  assurance  department 
c  in  u tr  oct  >1  p  i.  n  aid  n  y  d  y  s .  0  r.c  w  lisi  y  ..d-d  *  n  r  z  r y  id  * 

the  higher  in  the  data  processing  structure  the  quality 
assurance  function  reports,  the  better  the  probability  of 
success.  Also,  the  level  of  reporting  is  sometimes  indica¬ 
tive  of  management  support  [Ref.  4],  Figure  2.3  shows  quite 
a  simplified  view  of  a  representative  EDP  department  with 
the  qualify  assurance  function  placed  as  a  staff  function 
reporting  to  the  2DP  Manager.  This  structure  insures  tha^ 
quality  assurance  will  receive  the  attention  of  the  IT? 
Manager  and  that  it  will  be  independent  of  all  other  aspects 
of  the  data  processing  department.  Figure  2.10  shows  an 
organizational  structure  from  industry-  Informatics  Inc. 
Quality  Assurance  in  this  organization  is  embedded  in  the 
organizational  structure  and  his  secondary  affects 
throughout  the  company.  As  another  example.  Figure  2.11 
shows  a  variation  of  the  placement  of  the  quality  assurance 
function  [Ref.  25],  This  structure,  with  quality  assurance 
as  an  independent  function  reporting  directly  to  a  division 
general  manager,  provides  good  independent  oversight. 

Different  projects  within  an  organization  will 
receive  different  emphasis  in  the  quality  assurance  area. 
The  same  holds  true  for  industry  as  a  whole.  Embedded  soft¬ 
ware  in  a  weapon  system  will  not  receive  the  same  sort  of 
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Figure  2.9  Saeple  2DP  Department  Organization. 

quality  assurance  attention  as  a  payroll  project.  As  each 
project  has  differing  requirements,  so  will  each  quality 
assurance  scenario  be  different.  Throughout  literature 
concerning  quality  assurance  implementation,  there  are  a 
number  cf  methods  used  to  organize  a  quality  assurance  func¬ 
tion.  These  have  been  consolidated  intc  four  general 
methods  and  will  be  described  in  some  detail.  It  mus-  be 
remembered,  however,  that  the  type  of  organization  of  the 
quality  assurance  function  will  depend  greatly  cn  such 
factors  as  the  type  cf  product  and  industry,  emphasis  given 
quality  assurance  within  the  organization  and  the  scope  of 
the  project.  Quality  assurance  departments  from  selected 
companies  will  be  examined  in  detail  further  in  the  paper 
which  will  expand  on  the  methods  of  organizing  the  quality 
assurance  function. 

The  first  method,  and  probably  most  widely  used  when 
organizations  are  beginning  the  quality  assurance  function, 
is  the  task  force  method.  This  method  allows  organizations 
to  become  involved  in  some  sort  of  quality  assurance 
activity  prior  to  the  formalization  of  the  quality  assurance 
function.  A  task  force  offers  the  advantage  of  developing  a 
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Figure  2.10  Informatics  Inc.  Organization  Chart. 

group  able  to  handle  the  unique  problems  which  may  be 
encountered  in  a  given  software  project.  Task  force  members 
with  the  appropriate  background  can  be  hand-picked  by  the 
EDP  manager.  Another  benefit  is  the  training  afforded  the 
systems  designers  assigned  to  the  team  because  it  puts  them 
ir  a  position  of  analyzing  the  competency  of  systems  design. 
A  disadvantage  tc  this  method  is  the  problem  of  continuity. 
Each  task  force  will  tend  to  develop  its  own  methods  and 
procedures  for  the  review.  if  the  task  force  members  are 
not  relieved  of  a  significant  amount  of  the  burden  of  their 
daily  work,  then  another  possible  problem  is  that  they  may 
have  trouble  finding  adequate  time  to  devote  to  the  review. 


Figure  2.11  Traditional  Organizational  Structure. 

A  second  method  is  the  formation  of  a  full  time 
quality  assurance  staff.  This  method  provides  the  grea~esi- 
amount  cf  continuity  among  reviewers.  The  EDP  manager  can 
thus  have  a  greater  degree  of  confidence  in  the  quality 
assurance  function.  By  assigning  a  full  time  staff,  manage¬ 
ment  is  giving  a  signal  of  the  measure  of  importance  it 
places  on  quality  assurance.  The  biggest  disadvantage  cf  a 
full  time  staff  is  the  competency  of  the  review  group. 
Whereas  a  task  force  can  add  specialized  knowledge,  a  full 
time  staff  operates  with  the  personae!  assigned.  Another 
problem  is  the  technical  proficiency  of  the  staff. 
Technical  proficiency  with  current  practice  is  very  impor¬ 
tant  both  from  the  standpoint  of  credibility  of  the  staff 
with  the  rest  of  the  organization  ar.d  the  proficiency  with 
which  the  personnel  can  perform  their  function. 

The  permanent  committee  method  is  another  approach 
and  is  basically  just  a  step  up  from  the  task  force  method. 
Continuity  of  individuals  is  the  biggest  difference  between 
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the  tvc  methods.  Where  a  task  force  reviews  one  project,  a 
committee  will  be  convened  for  the  parpcse  of  reviewing  many 
projects.  This  says  to  project  managers  that  their  projects 
will  be  reviewed.  The  permanent  aspect  of  the  committee 
indicates  a  higher  degree  of  management  support  than  a 
specially  convened  task  force.  is  with  a  task  force,  a 
permanent  committee  has  the  problem  of  the  amount  of  time 
reviewers  can  devote  to  projects  under  review  while  still 
maintaining  their  workload.  Another  negative  aspect  is  that 
it  still  is  just  a  committee  and  will  lack  the  authorita¬ 
tiveness  of  a  function  staffed  with  full  time  personnel. 

The  fourth  method  is  a  comoinacion  of  full  time  and 
part  time  personnel.  This  method  car.  be  accomplished  with 
the  use  of  one  or  more  full  time  personnel  to  maintain 
continuity  of  the  quality  assurance  function  and  augmented 
by  part  time  personnel  to  assist  in  reviews.  The  obvious 
advantage  would  be  the  ability  to  aid  specialized  knowledge 
as  needed  to  review  projects.  If  this  method  is  used,  one 
individual  should  be  named  to  head  the  quality  assurance 
function.  He  should  be  a  strong  personality  with  supior 
knowledge  of  the  requirements  of  of  a  quality  assurance 
function  and  possess  the  ability  to  direct  part  time 
personnel  in  the  most  efficient  use  of  their  time.  It  is 
important  that  the  manager  be  on  equal  footing  with  other 
line  functions  and  can  insist  that  only  the  best  people  be 
assigned  to  quality  assurance. 

3.  Rev iews  and  Audits 

Reviews  are  conducted  sequentially  throughout  the 
lifecycle  in  order  to  facillitate  transition  into  a  subse¬ 
quent  developmental  phase.  As  previously  stated,  reviews 
should  occur  at  the  completion  of  each  development  phase. 
Perry  [Bef.  4]  has  further  identified  twelve  review  points 
which  will  not  only  review  but  influence  systems  design  as 
well.  These  reviews  occur  at  the  following  points: 
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1.  Sid  justif  ication  ph»  s®, 

2.  Tnd  of  justification  phase, 

3.  Business  system  solution  phase, 

4.  Computer  equipment  selection, 

5.  Computer  system  design, 

6.  Program  design, 

7.  Testing  and  conversion  planning, 

8.  Program  coding  and  testing, 

9.  Detailed  test  plan, 

10.  Test  results, 

11.  Detail  conversion  planning  anl  programs,  and 

12.  Conversion  results. 

Naturally,  the  number  of  review  points  would  depend 
on  a  number  of  variables  including  sire  of  system,  impact  or. 
the  organization,  and  makeup  of  the  quality  assurance 
organization.  In  addition  to  these  review  points,  quality 
assurance  can  perform  valuable  consultation  while  conducting 
the  reviews. 

Bos*  authors  are  not  as  specific  as  Perry  as  zo  the 
timing  and  placement  of  review  points.  The  general 
consensus  is  that  reviews  must  be  predefined,  occur  a*  key 
points  in  the  development  process,  be  understandable  and 
thorough,  and  are.,  conduot  ed  in  accordance  with  prescribed 
procedures. 

Thera  are  a  number  of  techniques  that  a  quality 
assurance  review  team  may  employ  during  the  course  of  the 
review.  When  gathering  information  about  the  system  being 
reviewed,  such  things  as  project  documentation,  system  docu¬ 
mentation,  interviews,  observations,  and  the  use  of 
established  checklists  are  appropriate  methods  of  gathering 
information.  Practices  used  when  attempting  to  validate  the 
information  given  during  the  gathering  phase  inculde 
testing,  evaluating  test  data,  foraulating  base  case  data, 
and  individual  confirmation.  After  the  information  is 
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Gathered  and  validated  quality  assurance  must  evalua*-  -.he 
data  fcr  management.  This  evaluation  is  typically  based  on 
intuitive  and  evaluative  judgement,  mathematical  simulation 
or  modeling,  expert  advice,  or  guantitative  analysis.  This 
list  is  not  exhaustive,  but  is  representative  of  the  types 
of  techniques  used  by  quality  assurance  reviewers  in 
achieving  fair  and  comprehensive  reviews. 

Auditing  is  sometimes  differentiated  from  reviews. 
Audits  are  usually  thought  to  be  final  acts  where  all  loose 
ends  in  a  quality  program  are  tied  up.  Types  of  audits 
include  in-house  audits  where  the  audit  verifies  that  the 
developer  is  adhering  to  all  development  standards  and 
procedures,  subcontractor  audits  to  ensure  than  the  subcon¬ 
tractor  is  complying  with  all  software  standards  and 
procedures  imposed  by  the  contract,  and  fact-finding  audits 
in  which  the  subcontractor  is  evaluated  to  ensure  he  is 
capable  of  furnishing  reliable,  quality  software  of  the  type 
deemed  necessary  to  meet  co ntracturai  requirements. 

The  Institute  of  Electronic  and  Electrical  Engineers 
standards  propose  a  certain  minimum  number  of  reviews  which 
should  be  conducted  during  the  software  development  life- 
cycle  [Ref.  7].  These  include  a  software  requirements 
review  tc  ensure  the  adequacy  of  the  requirements  stated  in 
the  software  requirements  specifications,  a  preliminary 
design  review  to  evaluate  the  technical  adequacy  of  the 
preliminary  design  of  the  software,  and  a  critical  design 
review  to  determine  the  acceptability  of  the  detailed  soft¬ 
ware  designs.  Recommended  audits  consist  of  a  functional 
audit  which  is  held  prior  to  software  delivery  to  verify 
compliance  to  all  requirements  specif ications,  a  physical 
audit  to  verify  that  the  software  and  documentation  are 
internally  consistent  and  ready  for  delivery,  and  in-process 
audits  to  verify  consistency  of  the  design. 
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4 .  Tasking 

Verification  is  another  word  for  testing.  It  is 
essentially  ensuring  that  the  conditions  are  as  stated.  It 
involves  doing  whatever  is  necessary  to  verify  that  the 
statements  or  conditions  are  correct.  Although  correctness 
is  the  overall  goal  for  most  testing  efforts,  it  is  not 
always  the  overriding  concern.  Large  programs  are  sometimes 
so  complex  that  they  never  completely  satisfy  their  specifi¬ 
cations.  These  programs  may  be  quite  usable  because 
failures  are  encountered  infrequently  in  practice,  and  when 
they  do  occur,  their  impact  on  a  user  is  small.  To  be 
usable,  correctness  is  not  always  necessary  and  sometimes 
not  good  enough.  A  correct  program  may  satisfy  a  narrowly 
drawn  specification  and  yet  not  be  suitable  for  operational 
use  because,  in  practice,  inputs  not  satisfying  the 
specification  are  presented  to  the  program  and  results  of 
such  incorrect  usage  are  unacceptable.  Thus,  if  a  program 
is  correct  with  regard  to  an  inadequate  specification,  its 
correctness  is  of  little  value.  The  problem  which  arises  is 
that  most  testing  consists  of  correctness  tests.  There  is 
very  little  testing  dene  for  reliability,  robustness,  effi¬ 
ciency,  and  other  properties  which  make  a  valuable  software 
product.  Whatever  property  is  being  tested,  the  tests  which 
are  valuable  are  those  where  the  result  is  not  predictable, 
so  that  application  of  the  test  and  acquisition  of  the 
result  constitutes  an  information  gain  or  a  reduction  in 
uncertainty  [Ref.  26].  To  achieve  this  goal,  tests  should 
check  the  program  at  the  boundaries  of  its  behavior.  In 
order  for  software  to  be  tested  in  the  most  efficient 
manner,  a  test  plan  with  complete  procedures  and  methodolo¬ 
gies  must  be  formulated.  Other  than  from  the  obvious  reason 
of  ensuring  test  efficiency,  the  reasons  for  this  are  to 
provide  an  audit  trail  of  testing  so  that  future  problems 
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may  be  dissected  from  the  point  they  initially  surfaced  and 
that  boundary  areas  where  testing  would  be  most  efficient 
may  be  easier  to  identify. 

Software  quality  assurance  should  become  involved  in 
testing  in  a  number  of  areas  as  illustrated  in  Figure  2.12 
Before  the  testing  begins,  it  should  ensure  that  all  soft¬ 
ware,  hardware,  and  the  environment  are  in  a  satisfactory 
state  ar.d  that  test  and  simulation  software  have  been 
defined  and  are  under  control.  rt  should  witness  loading 
and  running  of  the  software  and  ensure  the  test  results  are 
retained  and  all  discrepancies  noted.  Finally,  quality 
assurance  soould  ass  rst  j.me  runcorons  no  ensur-r  —  eterrma- 
tions  concerning  deviations  and  discrepancies  are  recorded. 

HIL-S-5 2 779  A ,  the  standards  for  software  development 
used  by  the  Department  of  Defense,  contains  a  comprehensive 
list  of  software  testing  procedures  [Ref.  22].  These 
testing  procedures  are  utilized  by  many  civilian  software 
development  organizations  and  consist  of  the  following:  (*) 
analysis  of  software  requirements  to  determine  testability, 
(b)  review  of  test  requirements  and  criteria  for  adequacy, 
feasibility,  and  traceability  and  satisfaction  of  require¬ 
ments,  (c)  review  of  test  plans,  procedures,  and 
specifications  for  compliance  with  contractor  and  contrac¬ 
tual  requirements  and  to  insure  that  all  authorized  and  only 
authorized  changes  are  implemented,  (d)  verification  that 
tests  are  conducted  in  accordance  with  approved  test  plans 
and  procedures,  (e)  certification  that  test  results  are  the 
actual  findings  of  the  tests,  (f)  review  and  certification 
of  test  reports,  (g)  ensuring  that  test  related  media  and 
documentation  are  maintained  to  allow  repeatability  of 
tests. 
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Before  Test  I 

1.  Ml  software  and  hardware  under  control 

2.  Ml  test  and  simulation  software  define1? 

3.  All  facilitias  available 

During  Test 

U.  Witness  loading  and  running 

5.  Record  discrepancies 

6.  Identify  and  retain  output  and  results 
After  Test 

7.  Participate  in  Analysis 

8.  Raise  outstanding  points  as  discrepancies 

9.  Retain  analysis  results 

10.  Certify  test  report  on  satisfactory  completion 


Figure  2.12  Typical  Functions  of  a  QA  Dept.  During  Testing. 


5.  Software  Quality  Assurance  Tools  and  Techniques 

Improving  software  development  and  test  processes 
depends  in  large  part  upon  the  application  cf  proper  tools 
and  techniques  to  the  development  lifecycle.  The  differen¬ 
tiation  between  tools  and  techniques  is  very  clear.  A 
technique  may  be  defined  as  a  procedure  for  implementing  a 
reliability  or  quality  goal.  Techniques  consist  of  stan¬ 
dards  and  procedures  used  in  development  and  maintenance  of 
software  systems  (Ref.  25].  Such  things  as  structured 
programming,  top-down  design,  system  modularity,  proper 
language  selection,  abstraction,  information  hiding,  and 
program  design  languages  are  generally  thought  to  be  techni¬ 
ques  in  achieving  software  quality. 

Tools,  on  the  other  hand,  have  been  defined  as  an 
automated  technique  (Ref.  25].  Computer  programs  which 
perform  measurement  tasks  which  would  otherwise  have  to  be 
done  manually  are  considered  tools.  There  are  a  large 
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number  cf  tcols  fo  ensure  software  quality  assurance  or  the 
marketplace.  The  problem  is  that  most  automated  testing 
tools  are  expensive  to  install  and  use,  different  taste 
require  very  different  tools  and  most  tools  are  incompatible 
with  each  other  (Ref.  27],  Selection  of  specific  tools 
should  be  done  only  after  careful  analysis  of  the  objectives 
desired  of  the  tools,  the  tools  cost-  funding  and  the  criti¬ 
cality  of  the  software  functions  to  be  tested.  Another 
consideration  should  be  the  phase  of  development  which  the 
software  is  in.  Figure  2.  13  shows  some  typical  tools  which 
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Figure  2.13  Tools  Used  by  Software  Phase. 

may  be  used  during  the  lifecycle  of  software  development. 
While  not  a  specific  list  of  tools  which  may  be  utilized, 
the  figure  gives  an  idea  of  the  types  of  tools  found  in 
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industry  today.  A  brief  explanation  cf  selected  tod-  fro® 
the  above  list  follows.  System  modeling — a  technique 
whereby  a  simulation  of  the  hard* are/sc ftware  system  i  = 
programmed  using  a  simulator.  Interactions  between  hard¬ 
ware,  software,  and  personnel  are  simulated  and  incompatible 
system  requirements  often  become  evident  after  system 
modeling.  PSL/PSA  (Problem  Statement  Language/Problem 
Statement  Analy z er)  -■'■This  is  a  specific  tool  licensed  by  the 
(Jniversity  of  Michigan,  Project  ISDOS.  It  provides  a  means 
for  describing  information,  computer  and  software  systems. 
A  requirements  data  base  built  from  several  contributers  can 
be  checked  for  consistency  and  formal  completeness.  Design 
modeling — Critical  algorithms  are  coded  in  a  representative 
manner  to  determine  of  the  design  will  result  in  the  desired 
accuracy  and  execution  times.  Timing,  memory  margins, 
resource  utilization,  and  traffic  rates  are  modeled  to 
ensure  adequacy.  Require® ents  traceability  tool — Software 
requirements  are  linked  to  successive  design  data  base 
entries,  test  planning,  and  rest  data  entries  to  provide 
requirements  traceability.  Interactive  debug  tocls — The 
debug  tool  controls  the  code  while  it  is  executed  and 
displays  memory  and  registers.  The  registers  and  memory  can 
be  displayed  while  the  code  is  executed  instruction  by 
instruction.  Preset  memory  locations  and  registers  hold  any 
desired  value  thus  allowing  branches  to  be  executed  and  the 
logic  debugged.  ICS  (interpretive  computer  simula¬ 
tion) — This  tool  allows  the  instructions ,  interrupts  and 
input/output  capabilities  to  be  made  visible  by  simulating 
the  architecture  and  memory  of  a  larger  computer.  The 
program  can  be  started  and  stopped  in  order  to  evaluate 
performance  of  the  program  at  various  points.  Stress 
tests— As  the  name  implies,  this  tool  tests  the  computer 
under  worst  case  conditions  of  various  parameters  such  as 
memory  input  rates,  memory  utilization,  etc. 
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Hardware/software  test  beds— *A  test  bed  is  the  system  level 
hardware  joined  with  the  system  software  and  combined  with 
the  appropriate  test  drivers,  monitors  and  environmental 
simulators  to  provide  as  near  an  operational  system  as  is 
possible.  Regression  testing — Regression  testing  uses  a 
standard  proven  test  for  testing  the  software  after  a  change 
has  been  incorporated  in  the  software  in  order  to  detect  any 
side  effects  or  errors  due  to  the  change.  These  tools  are 
but  a  few  of  the  literally  hundreds  on  the  market.  Tools 
can  be  a  valuable  and  useful  addition  to  any  software  devel- 
ooaent  life  cycle  but  must  be  chosen  carefully,  checked  out 
before  commitment,  and  used  in  the  proper  perspective.  This 
means  that  the  tool  should  be  recognized  as  a  tool  and  the 
results  should  be  evaluated  carefully  and  action  only  taken 
on  specific  results  generated  by  the  tool. 

6.  Software  Documentation 

Perhaps  the  weakest  link  in  modern  software  develop¬ 
ment  has  been  documentation.  There  are  a  number  of  sources 
of  information  which  provide  guidelines  concerning  software 
documentation.  MIL-S-52779A  and  the  1222  quality  assurance 
standards  both  provide  requirements  for  documentation. 
HIL-S-52779A  calls  for  all  documentation  standards  and 
programing  conventions  and  practices  to  be  used  for  all 
software  to  be  referenced  in  the  quality  assurance  plan. 
The  IEEE  standard  calls  for  identification  of  the  documenta¬ 
tion  governing  the  development  and  verification  of  the 
software  and  an  explanation  of  how  the  documents  are  to  be 
chacked.  It  further  calls  for  a  number  of  specific  docu¬ 
ments.  These  include  a  software  requirements  specification 
(SFS)  ,  software  design  description  (SDD),  and  software  veri¬ 
fication  plan  (SVP) .  PIPS  POB  38  provides  extensive 
guidance  concerning  documentation  of  computer  programs  and 
ADP  systems.  Software  documentation  is  an  extremely 
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critical  aspect  of  software  development  in  -hat  i~  is  -he 
means  cf  ccmmun ica tion  which  the  designer  uses  with  his 
colleagues,  management  and  the  technical  authorities  of  the 
customer.  Although  it  is  widely  recognized  that  good  docu¬ 
mentation  practices  should  be  maintained  for  ail  system 
projects  and  there  are  ample  guidlines  with  which  to 
proceed,  documentation  remains  largely  inadequate.  Much  of 
it  is  old,  it  is  poorly  written  or  written  in  such  a  manner 
as  to  be  incomprehensible  to  the  average  reader,  it  may  not 
be  thorough  or  leave  out  elements  which  are  critical  tc  the 
software  in  question,  or  it  hasn't  been  changed  tc  reflect 
current  practices.  software  quality  assurance  must  realize 
the  requirement  for  good  docu mentation  and  take  steps  to 
ensure  that  documentation  which  accompanies  developed  soft¬ 
ware  is  complete,  clear,  accurate  and  concise. 

7 .  configur  ation  Management 

Configuration  management  consists  cf  identifying  the 
configuration  of  the  software  system  at  discrete  Doints  ir. 
time.  The  purpose  is  to  systematically  monitor  changes  to 
this  configuration  and  maintain  the  integrity  and  trace¬ 
ability  of  this  configuration  throughout  the  system  life 
cycle.  It  is  primarily  concerned  with  ensuring  the 
integrity  and  continuity  of  design.  Quality  assurance, 
through  configuration  management,  should  enforce  the 
following:  (a)  Configuration  Identification- A  system  of 
recording  the  technical  description  of  individual  computer 
programs  and  supporting  documentation,  thus  documenting  the 
functional  and  physical  characteristics  of  the  configured 
item.  (b)  Configuration  Control-applies  to  configured  soft¬ 
ware  and  documentation  after  they  have  been  released.  It 
also  provides  a  control  for  changes  and  library  features, 
(c)  Configuration  Status  Accounting-the  recording  of  the 
status  of  the  system's  configuration.  The  purpose  is  to 
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know  exactly  what  the  currant 
was  achieved. 


configuration 
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III.  A  SORVBT  OF  MILITARY  AND  CIVILIAN  SO RARE  ^DALITY 

ASSURANCE  PROGRAMS 

A.  IHTRODOCTIOM 

As  stated  in  [Ref.  22]  the  purpose  of  a  quality  assu¬ 
rance  program,  "Is  to  assure  that  software  developed, 
acquired  or  otherwise  provided  under  the  contract  complies 
with  the  requirements  of  the  c outran t" .  Another  definition 
of  quality  assurance  is,  ''A  planned  and  systematic  pattern 
of  all  actions  necessary  to  provide  adequate  confidence  that 
the  items  will  perform  satisfactorily  in  actual  operation" 
[Raf.  28],  Still  another  definition  is,  "The  early  detec¬ 
tion  and  correction  of  deficiencies  and  the  evaluation  of 
overall  quality  performance"  [Ref.  29].  Although  pros  and 
cons  on  the  merits  of  each  of  these  definitions  car.  be 
thought  of  easily,  they  all  have  the  same  goal  -  prevent 
customer  complaints. 

Quality  assurance  achieves  this  goal  through  control. 
This  control  function  is  based  on  the  existence  of  some  type 
of  plan  and  the  control  function  is  simply  ensuring  adher¬ 
ence  to  that  plan.  Effective  control  will  detect  deviation 
from  the  plan  early,  before  it  actually  occurs.  Ineffective 
control  detects  deviation  a s  it  happens,  when  it's  too  late. 
Two  hey  factors  in  effective  control  are  (1)  total  knowledge 
of  the  plan  and  (2)  establishment  of  milestones  against 
which  progress  on  the  plan  can  be  measured.  By  monitoring 
these  milestones,  action  can  be  initiated  to  prevent  poten¬ 
tial  deviation  from  the  plan. 

In  essence,  quality  assurance  is  not  something  that  can 
be  added  later  in  the  software  development  process.  It  is 
not  the  job  of  a  single  person  or  group  of  persons  to  see 
that  quality  is  added  at  the  right  time  and  in  the  right 
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amount.  Quality  assurance  begins  at  the  s*ar-.  *f  rh  = 
development  process  and  is  continually  added  at  each  step 
along  the  way.  The  primary  job  of  the  quality  assurance 
group  then  becomes  the  development  of  the  quality  assurance 
plan  ar.d  once  it  is  developed,  the  management  cf  it 
throughout  the  development  process. 

The  next  sections  of  this  chapter  will  discuss  standards 
which  are  in  use  for  developing  quality  assurance  plans. 
Following  that  will  be  a  discussion  of  the  typical  software 
development  cycle.  The  final  section  of  the  chapter  will 
discuss  the  quality  assurance  programs  in  use  at  two  major 
civilian  companies  and  at  the  Naval  3cean  Systems  Center. 

B.  SOFTWARE  QUALITY  ASSURANCE  PLAN  STANDARDS 

1.  Military  Specificat ion  5?779& 

Mil -S-5  2779  A  applies  to  the  acquisition  of  software 
either  alcne  or  as  part  of  a  complete  system.  It  requires 
the  establishment  and  implementation  of  a  software* quality 
assurance  program  by  the  contractor. 

Paragraph  3.2  of  Mil-S-52779A  deals  with  the  soft¬ 
ware  guality  assurance  plan.  According  to  this  specif ication 
[Ref.  22]  any  software  quality  assurance  plan  will  include 
the  following  areas: 

1.  Tools.  Techniques  and  Methodologies:  What  are  they 

and  now  will  they  supDort  the  overall  Quality 
Assurance  Program?  Examples  include:  Operations 

Research  -  Systems  Analysis,  functional  and  perfor¬ 
mance  requirements  analysis,  error  analysis,  software 
optimization  tools,  spe cif ica tion  tracing  and  coding 
conventio  ns. 

2.  Computer  Program  pesign:  How  will  design  lcgjg, 
fulfillment  of  requirements,  completeness  and  comDii- 
ance  with  specified  standards  be  evaluated? 

3.  Work  Certification:  How  will  the  description,  author¬ 
ization  and  completion  of  work  be  certified  or 
approved  ? 

4.  Documentation:  What  documentation  standards  and 

program  conventions  will  be  used? 


54 


5.  Computer  Eroqrai  Library  Con-rols :  How  will  different 
comparer  orogram  versions  be  ider.tifi  =  d  i r.  1 
menied?  The  objective  here  is  to  insure  -ha*  only 
approved  modi  f  icatio  ns  ire  male  and  impiemer.*  e 

6.  Reviews  and  Audips:  How  will  reviews  and  audios  be 

conducted  to  insure  traceability  from  initial 
requirements  to  final  product? 

7.  Configuration  Management  (CM):  How  are  Software 

Quality  Assurance  and  CM  related? 

8.  Testing:  This  section  includes  the  following  areas  - 

a)  Analysis  of  requirements  to  determine  testability 

b)  Review  of  test  requirements  and  criteria  to  insure 
adequacy,  feasability,  traceability  and  satisfac¬ 
tion  of  requirements. 

c)  Review  of  test  plans,  procedures  and  specifica¬ 
tions  for  compliance  witn  contractual  :  -cuir  -rent? 
and  to  insure  all  authorized  changes  are  imple¬ 
mented. 

d)  Verification  of  tests. 

e)  Certification  of  test  results. 

f)  Review  and  certification  of  test  reports. 

g)  Maptenance  of  test  material  to  insure  repeat¬ 
ability. 

h)  Support  software  and  hardware  used  during  develop¬ 
ment  must  be  acceptable  to  the  government. 


2.  ITSS  Standard  for  Software  Quality  Assurance  Plans 

The  purpose  of  this  standard  is  to  provide  uniform, 
minimum  acceptable  requirements  for  preparation  and  content 
of  Software  Quality  Assurance  Plans.  The  standard  applies 
to  the  development  and  maintenance  of  critical  software 
(i.  e.  where  failure  could  impact  safety  or  cause  large 
financial  or  social  losses).  For  non-critical  software  or 
software  already  developed  a  subset  of  the  requirements  may 
be  used  [Ref.  23]. 

The  following  ace  the  major  sections  and  subsections 
of  the  plan  as  outlined  in  [Ref.  23]. 

1.  Purpose. 

2.  Reference  Documents. 

3.  Management. 

a)  Organization 
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b)  Tasks 

c)  Responsibilities 

4.  Documentation 

a)  Purpose 

b)  Minimum  Required  Documentation:  Software 

Requirements  Specification,  Software  Design 
Description,  Software  Verification  Plan. 

c)  Other:  Computer  Program  Development  Plan. 

Configuration  Management  Plan,  Standards  ana 
Procedures  Manual. 

5.  Standards,  Practices  and  Conventions. 

a)  Purpose 

b)  Content:  Document  Standards,  Logic  Structure 

Standards,  Coding  Standards,  Commentary  Stardrds. 

6.  Reviews  and  Audits, 
e)  Purpose 

b)  Minimum  Rquirements:  Software  Requirements  Review, 
Preliminary  Design  Review,  Critical  Design  Review, 
Functional  Audit,  Physical  Audit,  in-Prccess 
Audits. 

7.  Configuration  Management. 

8.  Problem  Reporting  and  Corrective  Action. 

9.  Tools,  Techniques  and  Methodologies. 

10.  Cede  Control. 

11.  Media  Control. 

12.  Supplier  Control. 

If  any  of  the  above  sections  are  not  pertinent  to 
the  project  for  which  the  plan  is  being  written,  a  statement 
stating  this  non -applicabil ity  should  be  included  under  the 
section  heading  along  with  reasons  why  it  is  not  applicable. 
If  additional  sections  are  needed  they  may  be  included  at 
the  end. 

C.  PHASES  OF  THE  SOFTWARE  DEVELOPMENT  PROCESS 

It  is  generally  agreed  upon  that  the  software  develop¬ 
ment  process  consists  of  at  least  the  following  sever,  phases 
[Ref.  24]  :  Conceptual,  Requirements  Definition,  Design, 
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Coding  and  Checkout,  Testing,  Integration,  ana  Operational, 
Software  definition  takes  place  in  the  Conceptual  phase. 
This  consists  of  feasability  studies,  trade-off  studies  and 
analyses  to  define  specific  requirements  to  be  allocated  to 
computer  resources.  Once  these  requirements  are  defined  and 
documented  they  form  the  basis  for  a  draft  system  specifica¬ 
tion  which  will  be  used  during  the  following  phases. 

During  the  second  phase.  Requirements  Definition,  it  is 
determined  which  system  requirements  will  be  implemented  by 
software.  Through  analysis  it  is  determined  which  software 
functions  are  needed  and  the  inputs,  processing,  and  outputs 
that  are  required  for  each  function.  Also  part  of  this  phase 
is  the  finalization  of  the  system  specification  and  the 
preparation  of  the  draft  software  requirements 
speci f ication. 

Following  the  Requirements  Definition  phase  is  the  third 
phase  of  the  development  cycle.  Design.  The  object  of  this 
phase  is  to  come -up  with  a  software  design  that  will  imple¬ 
ment  the  functions  identified  during  the  Requirements 
Definition  phase.  The  design  will  include  actual  algorithms 
and  equations  alcng  with  control  logic  an  data  operations  to 
be  performed.  The  finalization  of  the  software  requirements 
specification  and  the  preparation  of  the  draft  software 
design  specification  will  also  take  place  during  this  phase. 

The  fourth  phase.  Coding  and  Checkout,  includes  trans¬ 
lating  the  software  design  into  a  computer  programming 
language.  Usually  it  is  a  high-order  language  but  it  may 
alsc  be  assembly  language.  Once  compilation  and  assembly 
errors  are  corrected  each  individual  program  module  is 
executed  to  remove  obvious  errors.  This  procedure  consti¬ 
tutes  the  checkout. 

Once  coding  is  complete  the  fifth  phase  of  the  cycle. 
Testing,  begins.  Here  the  software  which  has  been  developed 
is  tested  to  shew  that  it  is  consistent  with  system  and 
software  requirements. 


i 
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During  Integration,  hardware  ar.d  software  are  crcMir.* 
together  and  system  and  operational  testing  is  conducted. 
The  object  cf  the  testing  is  to  insure  the  satisfaction  of 
system  requirements  in  the  actual  or  simulated  environment. 

The  last  phase  of  the  cycle  is  the  Operational  phase. 
The  software  has  been  accepted  for  use  and  the  only 
remaining  activities  are  maintenance  and  modification. 
Figure  3.1  [Bef.  24]  shows  the  software  development  process 


Figure  3.1  Software  Development  Process. 

and  the  key  outputs  of  the  phases.  Figure  3.2  [Bef.  24] 
shows  how  the  quality  assurance  activities  fit  into  the 
development  process.  Fiqure  3.3  [Ref.  29],  although  it  does 
not  specify  all  seven  phases  decribed  above,  it  dees  show 
the  activities  of  not  only  the  Quality  Assurance  group  but 
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Figure  3.2  Software  Quality  Assurance  Process. 

of  the  groups  in  the  prole ct  organization.  The  acriviries 
listed  are  all  guality  assurance  related. 


D.  QUALITY  ASSOBAHCE  PROS  HAMS 


1  •  III  Defense  Systems  Group 


a.  Background 


Kurt  F.  Fischer  [Ref.  30]  writes: 


Ap  TRW  Defense  and  Spape  Systems  arouo  the  need  for  a 
division  wide  organization  to  assist  software  management 
became  quite  apparent  during  the  Late  1960s.  Like  most 
other  software  vendors,  TRW  was  concerned  about  the 
frequent  cost  overruns  and  schedule  slippages  in  its 
software  projects,  and  decided  to  develop  methods  to 
turn  this  trend  around.  As  part  of  that  decision,  it 
established  its  first  quality  assurance  staff  in  1969. 
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Fiqure  J.3  Development  Ptocess  for  quality  Software. 


Figure  3.4  TRI  Corporate  Organizational  Structure. 


organizational  structure.  The  Product  Assurance  organiza¬ 
tion  actually  performs  a  dual  role:  Quality  Assurance  and 
Configuration  Management.  The  reason  for  this  is  that  both 
areas  have  been  found  to  share  similar  characteristics. 

1.  They  perform  staff  oriented  functions. 

2.  The  performance  of  .their  functions  is  often  times 
more  credible  when  done  by  an  independent  organiza¬ 
tion. 

3.  S-faff  personnel  share  m^ny  aptitudinal  characteris¬ 
tics  (e. g.  close  attention  to  detail,  preference  for 
wide  visibilty  tasks)  [Eef.  30]. 
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Product  Assurance  on  the  product  level  is  '-.eared 
by  dr.  Assistant  Program  .linages  for  Product  Assurar/o-  who 
has  responsibility  for  both  Quality  Assurance  and 
Configuration  Management.  The  APM  for  Product  Assurance 
receives  his  direction  from  the  project  manager,  yet  he  and 
his  staff  remain  organizationally  independent  from  the 
project  by  reporting  functionally  to  the  Product  Assurance 
organization  at  the  corporate  level.  Figure  3.5  [Ref.  30] 
illustrates  the  project  organizational  structure. 


b.  Quality  Assurance  Objectives 


To  achieve  the  seated  project  quail--/  assurance 

objectives,  the  following  activities  [Ref.  31]  are  performed 
by  the  Quality  Assurance  group  at  TRff : 

1.  Assure  direct  traceability  between  the  subsystem 
specif ications  and  the  de ve^to ment  specifications.  In 
addition  assure  direct  traceability  between  develop¬ 
ment  specifications  and  the  test  plan  and  from  there 
to  test  procedures.  QA  will  additionally  insure  that 
all  requirements  are  traceable  to  the  product  speci¬ 
fication. 

2.  QA  will  develop  and  maintain  a  software  retirements 
matrix.  This  matrix  will  be  maintained  throughout  the 
software  development  cycle. 

3.  Ail  documentation  generated  during  the  project  will 
be  reviewed  by  QA  personnel. 

4.  conduct  audits  of  the  software  development  process. 

5.  QA  personnel  will  participate  in  all  formal  reviews 

and  audits  (.e.g.  Software  Requirements  Review, 
Preliminary  Design  Review  and  Critical  Design 

Review) , 

6.  Insure  the  implementation  of  built  in  checks  and 
Balances. 

7.  QA  personnel  will  monitor  and  witness  the  Preliminary 

Qualification  Test  (PQD  and  Formal  Qualification 
Test  (FQT) .  These  test  results  will  be  cosianed  by  QA 

personnel.  All  QA  test  records  will  be  maintained. 

8.  QA  personnel  will  monitor  the  Configuration 

Management  practices  during  software  development. 

They  will  also  test  to  insure  the  integrity  of  the 
software  configuration. 

9.  QA  personnel  will  participate  in  all  Configuration 
change  Board  (CCB)  meetings  and  provide  a  review  of 
all  proposed  changes  in  the  software  development  and 
test  process  to  again  insurs  the  integrity  of  the 
software  configuration. 
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Figure  3.5  TRW  Project  Organizational  Structure. 


10.  QA  will  support  the  development  of  project  Software 
Standards  ana  Procedures. 

11.  Verify  that  all  requirements  and  functional  capabili¬ 
ties  nave  been  satisfied  By  software  testing. 

12.  QA  personnel  will  insure  that  the  delivered  software 
package  meets  contractual  requirements. 

13.  A  system  for  tracking  Software  Problem  Reports  (SPR) 
and  Design  Problem  Reports  (DPR)  throughout  the  soft¬ 
ware,  development  and  production  life  will  be 
developed  and  maintained  by  QA  personnel. 
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14.  To  insure  the  necessary  implementation  and  s utcor* 
for  all  Q A  and  CM  activities,  Quality  Assurance"  V ill 
Darticipate  in  the  selection  orocess'of  all  software 
^ools. 

c.  Quality  Assurance  Planning 

The  successful  implementation  of  an  effective 
Quality  Assurance  Program  relies  heavily  on  quality  assu¬ 
rance  planning  during  the  early  phases  of  software 

development.  At  TRW  QA  planning  is  accomplished  by  a 
complete  review  of  the  early  project  documentation.  Examples 
of  such  documentation  include  the  Contract  Statement  of 

Work,  System  Specifications  and  Pto-'sc*  Plan  among  others. 
Once  this  initial  review  is  completed  the  Quality  Assurance 
Plan  is  prepared.  This  plan  contains  the  functions,  tasks, 
and  responsibilities  of  the  Quality  Assurance  group  and  also 
identifies  the  quality  assurance  tools  needed  to  insure 
software  quality  in  the  areas  of  accountabilit y,  test¬ 
ability,  usability,  maintainability  and  reliability.  Upon 
completion  the  plan  is  reviewed  by  other  project  organiza¬ 
tions  ar.d  approved  by  both  the  Program  Manager  anc  the 

customer.  Upon  approval  task  assignments  are  made  to  carry 

out  the  activities  outlined  in  the  previous  subsection.  As 
noted  in  fRef.  30]  these  assignments  are  based  on  level  of 
effort  and  must  remain  flexible  to  adapt  to: 

1.  The  needs  of  the  current  phase  in  the  development 
life-cycle. 

2.  Shifts  in  attention  needing  areas  (.eg.  technical 
problems)  . 

3.  Unexpected  demands  placed  on  Quality  Assurance  by  the 
Project  Manager. 

Once  the  QA  Plan  is  approved  the  Quality 
Assurance  polocies  and  procedures  are  written  describing  the 
methods  and  procedures  to  be  used  in  the  implementation  of 
quality  assurance  requirements  defined  in  the: 

1.  System  Specification. 

2.  Contract  Statement  of  Work. 
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Project  Plan 
QA  plan. 


d.  Software  Star-lards 

As  Kurt  P.  Fischer  points  out  in  [Ref.  30], 

"The  purpose  of  software  standards  is  to  improve  the  main¬ 
tainability  and  readability  of  the  software  product".  TRW 
has  developed  a  comprehensive  and  detailed  program  to  deal 
with  software  standards.  According  to  [fief.  30]  this  program 
has  been  successful  for  two  reasons: 

1.  software  standards  are  not  dictated  from  the  execu¬ 
tive  offices  or  from  Quality  Assurance,  but  are 
developed  cut  of  close  communication  aacr.o  th  = 
Desicrn,  Development,  Test,  QA  and  Project  offices. 

2.  A  tool  has  b-e&  provided  to  automatically  check  the 
software  against  most  cf  the  standards.  This  allows 
programmers  to  audit  themselves  sc  that  there  are  no 
surprises  at  turnover  time. 

The  Software  Standards  and  Procedures  (SSP) 
[Ref.  31]  document  contains  the  qualify  provisions,  instruc¬ 
tions  and  standards  for  each  project.  The  SSP  deals  with 
standards  concerning  software,  firmware,  'design,  development 
ar.d  testing.  The  categories  of  standards  included  are: 

1.  Source  code  formatting  standards 

2.  Techniques  to  be  used  in  software/firmware  design, 
code,  test  and  update. 

3.  Standards  dealing  with  QA  tool  development  and  their 
use  during  design  development  and  checkout. 

Waivers  and  deviations  serve  to  complement  the 
standards.  They  allow  permanent  or  temporary  relief  from 
compliance  with  the  standards  due  to  technical  difficulties, 
inefficiencies  or  schedule  impact.  All  waivers  or  deviations 
must  be  approved  by  both  the  QA  Manager  and  the  Project 
Manager. 
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Quality  Assurance  Tools 


The  use  of  software  tools  should  only  be  consid¬ 
ered  where  it  will  prove  to  be  more  cost  effective  and  sore 
accurate  to  have  the  task  automated  rather  than  performing 
it  manually.  Tasks  which  may  fall  into  this  category  are 
often  tedious,  menial,  boring,  difficult,  error  prone, 
repetitive  and  costly  [Ref.  30]. 

Although  TRW  is  currently  using  improved  and 
updated  tools  the  following  are  examples,  taken  from 
[Ref.  30],  of  the  types  of  toois  in  use  at  TRW: 

1.  product  Assurance  Confidence  Evaluator  (PACE)  -  PACE 
is  designed  to  quantitatively  evaluate  how  tnercucilv 
and  rigorously  a  program  has  been  tested. 

2.  FORTRAN  Cede  Auditor  (FCA)  -  This  tool  audits  coding 
standards  and  by  doing  so  allows  enforcement  cf  these 
same  standards. 

3.  Structured  Programming  Auditor  (STRUCT)  -  STRUCT  is 

used  to  ensure  programs  comply  with  the  structured 
programming  standard.  It  is  executed  after  PACE 
because  it  relies  on  output  from  PACE.  It  evaluates 
program  structure  based  on  the  following  six 
constructs;  sequence,  if  -  then-else,  dc-while, 

do-until,  case,  esca pe-trom-ioop. 

u.  Units  consistency  Analysis  (UCA)  -  This  is  used  on 
FORTRAN  source  code  ar.a  the  associated  data  base.  It 
scans  the  code  and  interprets  equations.  Ev  refer¬ 
encing  the  data  base  for  the  variables  used*  in  the 
eguation  it  determines  if  the  units  in  the  assignment 
statement  are  in  fact  consistent. 


f.  Quality  Assurance  Reviews  and  Audits 


Review  and  audit  points  are  established  during 
the  QA  planning  phase  to  ensure  that  design,  code,  inspec¬ 
tion,  testing,  and  documentation  are  compatible. 

As  used  at  TRW  reviews  serve  as  quality  assurance  critiques 
of  documents  while  audits  are  critiques  of  processes. 
Evaluation  criteria  for  documents  [Ref.  30]  consists  cf  the 
following: 

1.  Adherence  to  format  and  pagination. 

2.  Clarity  of  objectives. 

3.  Technical  content. 
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4.  Ir.t  erdocu  ment  consistency. 

5.  Traceability  to  higher  level  specifications. 


The  audits  conducted  by  Quality  Assurance 
personnel  [Ref.  30]  serve  the  following  four  functions: 

1.  They  assess  compliance  of  source  code  and  documenta¬ 
tion  to  software  standards  and  procedures. 

2.  They  assure  traceability  of  requirements. 

3.  They  determine  the  satisfaction  of  system  require¬ 
ments  during  system  test  and  acceptance  phase. 

4.  They  assess  test  sufficiency. 

The  following  list  contains  the  audits  conducted 
by  the  quality  assurance  personnel  at  TRW.  A  brief  descrip¬ 
tion  of  each  audit  is  also  included. 

1.  Unit  Development  Folder  (UDF)  Audits  -  The  ODF  is  a 
nor.- deliverable  item  which  provides  a  'nechar.is’n  for 
internal  control  and  also  provides  manage aer.-  visi¬ 
bility  for  software  development.  The  UDFs  are 
prepared  and  maintained  for  each  software  unit 
provided  by  the  project.  The  definition  of  a  unit  is 
a  single  routine  or  a  grouD  of  looically  related 
routines.  The  UDF  is  a  collection  of  all  require¬ 
ments,  design  data,  code,  and  test  data  pertaining  to 
a  specific  unit.  The  UDF  serves  as  the  d:’  marv 
surveillance  mechanism  for  the  ouality  assurance 
personnel  durina  the  aasian,  coaina  ana  unit  test 
phases  .of  the  software  development  project.  Each  UDF 
ls  2, ud.i.'t s im  This  iudi*  is  ir.do  ohdsss 

in  an  effort  *to  pro vlde~early  detection  a  r.d  correc- 
tion  of  possible  errors.  Each  phase  is  designed  to 
audit  a  specific  irca  of  the  development  process. 
Following  is  a  description  [Ref.  31]  cf  each  phase: 

a)  Phase  I  -  verifies  appropriate  UDFs  have  been 
initiated,  proper  cover  sheets  and  inserts  have 
been  included,  requirements  nave  been  stated  in 
the  requirements  section  and  that  the  design 
section  contains  the  current  working  design. 

b)  ?hase  H  -  A  desk  check  and  automatic  code  audit 
is  performed  to  determine  if  code  isbeir.g  produced 
in  accordance  with  established  project  standards 
and  guidelines. 

c)  Phase,  III  -  Verifies  that  each  UDF  audited 

contains  a  compilation  of  test  results  and 

analyses  necessary  to  demonstrate  that  the  unit  of 
code  has  been  debuaged,  j nit  development  testing 
is  complete  and  that  ar.  UD-to-data  design  descrip¬ 
tion  exists. 


2.  Test  Data  Folder  (TDF)  -  The  TDF  is  the  primary 
working  document  for  the  test  yrouo.  As  such  It  also 
provides  a  surveillance  vehicle  for  quality  assurance 
personnel.  During  *he  integration  and  acceptance 
testing  phases  of  the  development  project.  The  TDF 
contains  the  test  requirements,  test  plan,  test 
procedures,  test  execution  reports  and  software 
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problem  reports.  Quality  Assurance  reviews  the  fclder 
tc  insure  edecuacv  ccm plezer.e ss  and  co".i''n,-''c  to 
standards  outlined  ir.  the  Software  Standards  and 
Procedures  (SS?)  . 

з.  Configuration  Management  Audits  -  Quality  Assurance 
uses  thus  $udit  to  monitor  the  configuration  manage¬ 
ment  activities  to  insure  that  they  comply  with  both 
the  CM  plan  and  documented  procedures. 

и.  Interface  Verification  Audit  -  This  audit  is 
conducted  as  early  as  possible  in  an  effort  to  iden¬ 
tify  and  correct  possible  interface  problems.  QA 
personnel  examine  the  requirements  design  and  oroarara 
specifications. 

5.  Preliminary  and  Detailed  Design  Audits  -  Conducted 
prj.op  to  the  preliminary  Design  Review  (PDR)  ana  the 
critical  Design  Review  (CDRf  respectively,  these 
audits  are  concerned  with  the  forma*  and  content  of 
design  documentation  and  Droject  test  plans.  The 
results  of  these  audi*s  are' t  h  *  t  discussed  a*  *hc  Dn" 
and  CDR  respectively. 

6.  Independent  Quality  Audit  -  N'ot  only  do  the  Product 
Assurance  personnel  audi*  the  various  areas  of  the 
software  development  crc'sott  but  they  in  turn  are 
audited  by  the  corporate  'Proiuct  Assurance  organiza¬ 
tion.  This  audit  does  in  fact  cover  the  whole 
project,  however,  the  QA  and  CM  areas  are  examined  in 
detail.  Upon  completion  of  the  audit  both  the  Project 
Manager  and  Assistar.*  Project  Manager  for  Product 
Assurance  receive  copies  of  the  audit  report.  They 
also  receive  any  Corrective  Action  Requests  (CAR) 

•  which  document  any  d iscrepanci es  found  by  the  audit. 

7.  Aud^t  Reports  -  The  finainos/results  of  the  above 
audits  are  provided  to  bota'ths  Project  Manaoer  and 
the  Assistant  Project  Manager  for  the  appropriate 
development  area.  In  addition  to  the  fin  dinas  recom¬ 
mendations  are  also  included.  A  periodic'  summarv 
which  details  the  number/type  of  audits  performed^ 
the  discrepancies  found,  all  corrective  action  in 
progress  cr  implemented  and  a  follow-up  on  corrective 
action  on-going  from  previous  reports  is  also 
prepared  by  Quality  Assurance  and  provided  tc  the 
Project  Manager. 

As  previously  noted  at  the  beginning  of  this 
subsection  reviews  are  conducted  to  critique  documentation. 
What  follows  is  a  list  of  the  reviews  conducted  at  TRW.  A 
brief  description  of  each  review  is  also  provided. 

1.  Software  Requirements  Review  -  This  review  is 
conducted  upon  completion  of  the  Sorhware 
Requirements  Definition  phase.  QA  personnel  sit  as 
members  of  the  review  board.  The  purpose  of  the 
review  is  to  insure  that  the  software  requirements 
specifications  for  the  proposed  software  project  do 
in  fact  match  the  users  operational  requirements  for 
the  system. 

2.  Preliminary  Design  Review  -  Ip  addition  to  conducting 
the  Preliminary  Design  Audrr,  Quality  Assurance 
personnel  act  as  recording  secretary  during  this 
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review.  The  PDR  TRef.  31]  reviews  each  deve ferment 
specification  for  the  following : 

a)  Traceability  of  requirements  specification  to  the 
development  speci fica tion. 

b)  Requiremnt^s  satisfaction,  interface  definition 
and  specification  content. 

3.  Critical  .  Design  Review.-  in  addition  to  performing 
the  Detailed  Design  Audit  Quality  Assurance  personnel 
serve  as  recording  secretary.  The  emphasis  or  the  CDR 
is  on  the  preparation  of  required  materials,  brief ina 
content  and  tne  allocation  of  each  requirement  to  a 
funcriona 1  design  element  [Ref.  31], 

4.  Design  Walkthroughs-  These  walkthroughs  are  held 
early  and  oftenduring  the  Design  phase.  The  walk¬ 
throughs  are  conducted  by  technical' personnel  who  are 
taken  through  the  design  on  a  stip-by-step  basis 
informally  by  the  designer.  This  done  in  an  effort 
to  monitor  o he  r ons isos r.oy  of  oh=  l-rsior.  approach, 
satisfaction  of  requirements  and  completeness 
[Ref.  31]. 


g.  Testing 


In  [  Bef .  31]  it  states  that  Quality  Assurance 
has  review  authority  for  all  rest  plans  and  procedures 
initiated  b7  TRW  for  either  formal  or  informal  testing.  QA 
performs  a  selective  review  of  documentation  to  insure  that, 
test  cases  and  rest  procedures  directly  correlate  wioh  the 
software  requirements.  The  two  primary  functions  performed 
by  Quality  Assurance  dur  ig  the  Testing  phase  are  test 
auditing  and  the  inspection  and  surveillance  of  formal 
tests.  Both  these  areas  are  described  below. 

1.  Test  Audit  -  This  a  gdit  is  conducted  af  the  end  of 

each  test  phase  and  its  primary  purpose  is  to: 

a)  Assure  that  software  configuration  management 
procedures  are  being  followed. 

b)  assure  that  the  test  specifications  beino  u$ea  by 
the  test  group  are  the  current  approved  versions. 

c)  Assure  that  test  reports  identify  proper  test 
procedures  ?ind  software  configuration;  specify  the 
test  analysis,  and, if  any  deficiencies  were  noted 
how  they  were  explained  and  accounted  for. 

d)  Verify  that  test  procedures  provide  a  step-by-steD 
rationale  for  cpnductmg  a  test  and  that  t$st 
results  comply  with  acceptable  criteria  specified 
in  the  test  procedure. 

e)  Verify  that  Test  Data  Polders  comply  with  approved 
f  or  ma  h  s. 
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3. 


a. 


f)  Verify  compliance  hy  the  ies‘  team  wi-.;.  : 

manaaement  procedures  for  change  control,  ii 
pancy  reporting,  and  rest  reporting  [Bef.  30] 
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Inspection  and  Surveillance  of  Formal  Tests  -  This  is 
an  on-site  activity  performed  by  QA  personnel  during 
program  testing.  Tne  purpose  :s  to: 


a)  Monitor  all  tests  to  ensure  that  the  actual  tests 
performed  are  those  specified  in  the  documented 
test  procedures. 

b)  Assure  all  potential  discrepancies  are  recorded  in 
the  approved  manner. 

c)  compare  configuration  of  hardware/software 
compents  used  m  the  test  against  the  configura¬ 
tion  identified  in  the  etst  procedures. 

d)  Certify  that  analysis  of  test  results  is  correct, 
the  test  satisfies  the  intended  requirements  and 
acceptable  criteria  and  the  Test  Data  Folder  is 
complete. 

e)  Assure  that  master  conies  of  test  procedures,  test 
results  and  test  reports  are  maintained  and  avai¬ 
lable  from  a  centralized  records  center  (Bef.  30]. 


Test  Changes  -  The  test  director  may  make  corrections 
of  typographical  errors  in  the  test  parameters  as 
long  as  they  do  not  deviate  from  specified  require¬ 
ments,  he  may  also  make  changes  in  the  test  set-up 
and  ir.  addition  he  may  make  changes  in  the  procedure 
step  sequence  provided  that  test  parameters  or  toler¬ 
ance  accuracies  are  not  changed.  All  changes  must  be 
documented  and  will  be  -reviewed  for  approval  by  the 
Test  Review  Board  and  Configuration  Control  Board. 
Any  changes  causina  a  deviation  from  specified 
requirements  must  oe  submitted  through  both  the 
Configuration  Control  Board  and  tne  customer 
f Ref.  31 ]. 


Test  Chan  ge .  Request  (TCR)  -  This  is  the  vehicle  u§ed 
for  requesting  changes  to  test  procedures.  A  descrip¬ 
tion  of  the  problem  and  the  proposed  changes  are 
included  cn  the  TCR .  Upon  approval  by  the  CCB  it 
provides  both  the  solution  to  the  problem  and  the 
means  for  implementing  that  solution.  Figure  3.6 
(Bef.  32 ]  is  a  sample  TCP  used  at  TEW. 


fc.  Problem  Reporting  and  Review 


In  addition  to  the  problem  reporting  and  review 
procedures,  the  procedures  used  at  TRW  for  the ‘control  of 
changes  to  the  scftware  product  will  also  be  discussed.  The 

change  control  procedures  fit  in  at  this  point  because  more 

often  then  not  changes  are  made  in  response  to  problems 
which  arise  during  the  development  process. 

1.  Configuration  Control  Board  (CCB)  -  At  TRW  the 

Configuration  Control  Board  has  been  established  to 
review  and  approve  changes  to  documentation  and 
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software.  Regularly  schedule!  nestings  are  hell  hut 
if  circu  a  stances  warrant  it  the  Project  Manager  can 
call  a  special  meeting.  The  project  manager  acts  as 
chairman  and  members  include  bersor.nel  frcra  Product 
Assurance,  Data  Processing  Systems  Fr  air.^rina. 
Applications  Software,  Systems  and  Support 'Software. 
Data  Processing  Hardware,  and  Integration  ana 
Testing.  The  user  may  also  take  part  in  the  meetings 
and  in  any  event  will  be  supplied  a  copy  of  the 
minutes. 

The  following  are  the  documents  used  at  TRW  in 
an  effort  to  maintain  configuration  control  over  a  software 
development  project.  These  documents  are  submitted  to  the 
CCB  for  its  review  and  approval. 

1.  Desicrn  Problem  Report  (D?R)  -  This  report  is  used  to 

reguest  changes  to  baselined  {already  approved)  docu¬ 
ments  and  also  to  initiate  chanaes'  to  formally 
baselined  customer  conrroll=i  documenrs.  The  DPR 
contains  both  a  description  of  the  problem  and  a 
proposed  solution.  Once  it  has  been  approved  by  the 
Configuration  Control  Board  it  provides  •‘■he  accepted 
solution  to  be  implemented.  Harked  up  change  pages 
must  be  attached  to  the  DPR  to  illustrate  rhe  chanaes 
being  made.  QA  personnel  monitor  the  resolut  ion '  of 
all  DPRs  [Ref.  32].  Figure  3.7  [Ref.  32]  is  an 
example  of  the  DPR  used  at  TRW. 

2.  Software  Problem  Report  (SPR)  -  The  SPR  is  a  request 
for  permanent  changes  to  internally  controlled  code. 
It  includes  a  description  of  the  Drcbiem.  identifies 
the  library  and  routines  affected  and  proposes  a 
problem  solution.  Once  approved  by  the  CCS  it  serves 
as  an  authorization  to  update  the  master  library 
[Ref.  32  1.  Fiaurs  3.3  [Ref.  32]  is  an  example  of 
TRW's  Software  Problem  Report. 

3.  Temporary  Modification.  Notice  (TMN)  -  The  TMN 
requests  and  implements  temporary  changes  :c  base¬ 
lined  code.  Included  with  the  TMN  are  a  listing  of 
actual  changes,  reasons  for  the  change,  any  restric¬ 
tions,  testing,  verification  and  files  affected.  The 
TMNs  are  correlated  to  SPRs  which  implement  the 
permanent  change  [Ref.  32].  Figure  3.9  [Ref.  32]  is 
an  example  of  a  Temporary  Modification  Notice. 


Bene  fits 


Kurt  F.  Fischer  notes  in  [Ref.  30]  that  TRW 
received  the  following  benefits  through  the  implementation 
of  its  QA  Program: 

1.  It  has  provided  increased  management  visibilty  into 
the  development  process  through  reviews  and  audits. 

2.  Project  risk  has  been. reduced  thr<?ugh  better  require¬ 
ments  traceability  and  more  disciplined  and  thorough 
testing. 

3.  It  has  enforced  software  standards. 
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4.  The  development  and  maintenance  of  software  rods 
beer,  centralized. 

5.  Quality.  Assurance  records  have  been  generalized. 
These  include,  probj.em  reports.  deviations  and 
waivers,  reviews  and  audits,  and  test  and  inspection 
reports  among  others. 

6.  A  skill  center  for  personnel  with  multi-pro iect  visi¬ 
bility  whc  are  better  able  to  prepare  project  plans 
and  procedures. 

7.  An  independent  group  assuring  that  deliverable  items 
meet  contractual  requirements. 


j.  Lessens  Learned 


Kurt  Fischer  states,  "Probably  the  largest 
lesson  learned  is  that  one  key  to  the  successful  development 
of  software  is  the  employment  of  a  strong  QA  activity", 
[Hef.  30].  In  addition  to  the  above,  Fischer  points  out  in 
[Bef.  30]  that  the  following  lessons  were  learned  by  TRW 
during  the  implementation  of  its  QA  Program: 

1.  Insure  adequate  QA  participat'd  on  durina  the  proposal 
and  contract  definition  phases. 

2.  Hiring  personnel  knowledgeable  in  software  and  then 
training  them  m  quality  assurance  is  easier  than 
hiring  QA  people  and  training  them  in  software. 

3.  Perform  the  first  audit  earlv  in  the  development 
proc*»ssto  allow  plenty  of  time  for  corrective  action. 

4.  Announce  the  audit  well  before  it  takes  place.  The 
object  is  to  ensure  it  was  done  rioht,  net  to  find 
problems . 

5.  Construct  an  audit  checklist  and  distribute  it  to  the 
area  being  audited  at  the  same  time  the  audit  is 
announced.  This  eliminates  subjective  assessments  by 
the  auditor  and  informs  the  Darty  being  audited  of 
the  exact  scope  and  depth 

of  the  audit. 

6.  Assign  QA  engineers  on  along  term  basis  so  that  they 
may  develop  a  relationship 

with  each  project  sub-group.  QA  engineers  should 
colocate  with  tne  development 
personnel  so  that  they  might  better  understand  the 
problems  of  other  project  members. 
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DISPOSITION  CC»  ACTION  1  PROBLEM  I  SUBJECT 


2 .  Naval  Ocean  Systems  Cen t er 
a.  Background 

The  Naval  Ocean  Systems  Center  (NOSC)  estab¬ 
lished  its  software  Quality  Assurance  program  to  assist 
procuring  activities  in  enquiring  quality  software.  Quality 
software  is  defined  as  software  which  meets  all  requirements 
of  operability,  reliability,  and  maintainability.  The 
expressed  mission  of  the  Software  Quality  Control  organiza¬ 
tion  is  to  provide  assistance  to  project  managers  in  the 
acquisit ion/mana gement  of  higher  quality  software  products 
through  ■‘■he  iaipl erne r. tat icr  of  certain  standard  practices.  It 
provides  a  manageable  structure  to  the  software  development 
process  through  document  inspection,  configuration  manage¬ 
ment,  and  testing  support.  Standard  techniques  include 
inspecting  and  evaluating  the  documentation  of  computer 
systems,  evaluating  a  projects  configuration  management 
procedures  in  regards  to  software  documentation  and  computer 
programs,  assuring  the  integrity  of  tested  programs  through 
program  library  control,  increase  user  confidence  through 
testing,  and  being  an  active  participant  on  project  change 
Control  Boards. 

The  activities  of  the  Software  Quality  Control 
organization  are  geared  to  the  projects  life-cycle.  This  is 
true  whether  the  project  is  short,  requiring  a  minimum 
effort,  or  long,  extending  over  a  multi-year  period.  Because 
of  this  fact  each  SQC  step  is  tied  to  the  project  plan, 
milestone  schedule,  and  list  of  configuration  items  expected 
to  be  baselined  [Ref.  33  ]. 

At  N CSC  the  Quality  Assurance  office  has  been 
established  at  the  directorate  level.  Figure  3.10  [Ref.  33] 
shows  the  NOSC  organizational  structure. 
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Figure  3.9  Temporary  Modification  Hotic 
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Figure  3.10  NOSC  Organizational  structure. 


b.  Quality  Assurance  Objectives  ar.d  Policies 


obj  ectives 


The  following  are  the  most  significant  of  zh' 
established  for  the  Software  Quality  Assuran: 
organization: 

1.  Ensure  consistency  of  software  baseline  development. 
Ensure  compliance  with  design  standards. 

Ensure  compliance  with  programing  standards. 

Ensure  adequacy  and  completeness  of  testing. 

Review  all  test  results.  [Ref.  33] 


2. 

3. 

4. 

5. 


73 


The  following  policies  hive  been  established  at 
NOSC  in  regards  to  Software  Quality  Assurance: 

1.  The  developing  directorate  has  the  basic  rc-s sensi¬ 
bility  for  the  quality  of  products  delivered  by' NOSC. 
Each  Director  shall  utilize  the  established  quality 
assurance  resources,  as  appropriate,  to  assure 
adequate  quality  of  all  end  products. 

2.  The  Director  of  the  Engineering  and  Computer  Sciences 
directorate  acts  as  Center  managements  agent  for 
product  quality  assurance  on  all  Center  projects.  As 
such  he  will  keep  Center  management  informed  of  the 
results  of  reviews  and  audits  cf  products  developed 
and  produced  by  the  center. 

3.  The  Quality  Assurance  office,  which  reports  directlv 

to  the  Director  of  the  Engineering  and  Computer 

Sciences  directorate,  will  be  the  point  of  contact 
for  the  coordination  of  all  Center  quality  assurance 
activities.  The  QA  office  is  responsible  for  keeping 
th«  Center's  QA  policies  current  and  responsive  to 
higher  Navy  directives  and  instructions,  (fief.  33] 

c.  Quality  Assurance  Planning 

The  Software  Quality  Control  Plan  is  prepared  by 
SQC  personnel  through  interviews  conducted  with  the  Project 
Manager.  The  plan  is  coordinated  with  project  plans  and 
provides  a  description  of  how  the  elements  of  quality 

management  will  be  applied  to  the  project.  Each  SQC  plan  is 
tailored  to  project  requirements. 

For  the  Quality  Assurance  program  to  be  effec¬ 
tive  the  SQC  plan  must  contain  certain  elements.  These  are: 
planning  for  products  at  the  end  of  each  task  using  manage¬ 
ment  audits  and  reviews  as  a  measure  of  completion  of  the 

products,  developing  documentation  in  the  proper  sequence, 
and  accepting  SQC  inspection  assistance  to  the  Project 
Manager  as  an  aid  in  ensuring  successful  system  turnover 
[Hef.  33]. 

d.  Software  Standards 

Software  Quality  Control  personnel  are  respon¬ 
sible  for  developing  and  enforcing  design  and  programming 
standards.  In  the  area  of  design,  standards  dealing  with 
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clarity,  derail,  logical  efficiency,  technical  maturity,  and 
consistency  with  functional  specifications  must  be  enforced. 

Programming  standards  provide  the  required 
consistency  in  the  technique  and  processing  required  for 
continued  software  support  throughout  the  system's  life- 
cycle.  The  programming  standards  deal  with  the  following 
areas:  logic  and  coding  conventions,  flow  chart  standards, 
intermodule  com m unications ,  programming  language  structure 
and  use,  data  design,  module  segmentation,  and  logic  error 
checking. 

Quality  Assurance  personnel  review  the  program¬ 
ming  effort  with  regard  for  compliance  with  standards.  If 
any  non-compliance  is  found  they  will  take  steps  to  ensure 
conformity  with  *:he  standards. 

e.  Quality  Assurance  Reviews  and  Audits 

The  software  review  process  exists  so  that  a 
qualified- decision  may  be  made  to  recommend  advancement  from 
one  phase  to  the  next.  Audits  on  the  other  hand  are 
conducted  to  verify  configuration  items  conform  to  specifi¬ 
cations  and  other  contract  requirements.  The  results  of 
reviews  and  audits  are  reported  directly  to  the  development 
directorate  by  quality  assurance  personnel. 

In  addition  to  the  reviews  and  audits  the 
process  of  baselining  will  be  discussed.  It  is  included  here 
because  it  is  an  integral  part  of  the  audit  and  review 
process. 

Baselines  are  a  configuration  management  tech¬ 
nique  used  to  control  the  development  of  a  software  product, 
as  stated  in  [Ref.  3U], 


A  project?  software  configuration  is  the  prevailing 
stare  of  its  softwar  a  compon?nts.  Those  components 
which  are  subjected  to  systematic  management  are  termed 
Configuration  Items  (CIs»  .  Software  CIs  are  qualified  as 
being  elements  of  an  evolving  software  product  which  are 
set  forth  in  technical  documentation  (including 
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s  cecificatior.s ,  drawings ,  and  listings)  ,  and  achieve-: 
the  computer  program  itself  (resident  or.  card,  tape, 
disk)  . 


In  [Bef.  33]  it  goes  on  to  point  out. 


Baselines  are  employed  throughout  the  life-cycle  of  a 
configuration  item  to  ensure  orderly  transition  from  one 
major  commitment  point  to  the  next  in  the  system  engi¬ 
neering,  software  development  production,  and  logistic 
support  processes.  Baselines  are  established  at  those 
points  m  a  program  where  it  is  necessary  to  define 
formal  departure  points  for  control  of  future  changes  in 
performance,  design,  development,  production,  and 
related  technical  requirements. 


he  baseline  is  created  upon  acceptance  of  a  de velor men'' 


document  or  product. 

There  are  four  types  of  baselines  used  at  NOSC, 
Functional,  Allocated,  Product,  and  Operational  fHef.  34]. 
A  brief  description  cf  each  follows. 

1.  Functional  Baseline  -  This  considered  the  hiahest 
level  baseline.  The  technical  documentation  ai'this 
level  delineates  all  the  necessary  functional  charac¬ 
teristics,  the  tes*s  which  will  be  required  to 
demonstrate  their  achievement,  all  necessary  inter¬ 
faces,  and  any  and  all  design  constraints.  This 
baseline  generally  covers  a_i  the  documentation 
produced  pror  to  develooment  of  the  software  design 
ar.d  the  formal  System  Design  Review. 

2.  Allocated  Baseline  -  At  this  the  next  lower  level 
baseline,  the  performance  oriented  specif icationst 
which  are  subordinate  to  the  Configuration  I^.ems  or 
the  functional  level,  expand  upon  allocated  func¬ 
tional  characteristics.  All  documentation  produced 
short  of  the  Preliminary  Design  Review  is  covered  by 
this  baseline. 

3.  Product  Easeline  -  This  is  cogsideped  the  lowest 
level.  The  documentation  at  hhis  point,  which  is 
subordinate  to  both  the  functional  and  allocated 
levels,  defines  the  production,  operation,  mainte¬ 
nance,  and  logistic  support  phases  of  its  life-cycle. 
This  normally  covers  all  documentation  and  proorams 
produced  prior  to  the  Formal  2ualif ication  Review. 

4.  Operational  Baseline  -.One?  the  .developed  system 
passes  the  Formal  Qualification  Review  and  has  proved 
it  meets  operational  requirements,  the  operational 
baseline  is  established.  AIL  modifications  required 
to  the  system  during  its  life-cycle  will  be  performed 
from  this  baseline. 
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The  baselining  technique 


TR  W  d  u  r  i  n  a 


software  development  process. 

Following  is  a  list  of  the  reviews  conducted  by 
the  Software  Quality  Control  organization  at  NOSC.  A  brief 
description  of  each  review  is  included. 

1.  Initiation  Review  -  The  purpos?  of  this  review  is  go 
affirm  the  Operational  Requirement  as  the  basic 
guideline  for  the  project.  Prior  to  the  review  the 
Operational  Requirement  should  have  been  read  by  all 
members  cf  the  project  team  including  software 
quality  assurance  personnel.  The  operational 
Requirement  is  reviewed  to  ensure  no  requirements  or 
constraints  have  been  omitted  [Ref.  33]. 

2.  Systems  Requirement  Review  -  The  objective  of  this 

review  is  to  determine  the  adequacy  of  the  develop¬ 
er's  efforts  in  definina  system  rsauir emer.t s .  The 

review  is  conducted  or.ce  a  significant  portion  of  the 
system  functional  requirements  have  been  established 
[Pef.  33]. 

3.  Document  Review  ana  Substantiation  -  This  occurs  in 

each  phase  of  the  development  life-cycle.  The  orimarv 
concern  is  to  review  draft  documentation  for 

completeness  and  correctness.  Quality  assurance 
personnel  inspect  and  substantiate  the  contents  of 
all  documentation.  The  documentation  under  review  is 
compared  against  established  standards  to  ensure  it 
contains  the  proper  level  of  detail  and  that  all  the 
required  content  is  present.  ‘lot  only  is  a  sin-ole 
piece  of  documentation  reviewed  by  itself  but  it  is 
compared  to  all  associated  documentation  to  ensure 
completeness  and  consistency.  All  deviations  from 
standards  and  any  technical  problems,  are  noted  and 
submitted  to  both  the  Project  Manager  and  the  devel¬ 
oper  for  correction.  Once  the  Operational  Requirement 
has  been  reviewed  and  approved,  the  remainder  of  the 
documentation  produced  by  the  project  is  reviewed  to 
ensure  it  is  consistent  with  and  meets  the  require¬ 
ments  set  forth  in  the  Operational  Requirement 
[Ref.  33]. 

u.  System  Design  Review  -  Once  the  project  team  has 
determined  that  the  software  requirements  documents 
fulfill  all  requirements  and  present  a  suitable  allo¬ 
cation  of  performance  requirements  between  hardware, 
software,  and  human  actions,  the  System  Design  Review 
is  held.  Also  the  identification,  correlation, 
completeness,  and  risk  of  the  software  requirements 
documents  is  evaluated.  Upon  approval  these  documents 
are  considered  baselined  [Ref.  33]. 

5.  Preliminary  Desian  Review  (PDR)  -  The  primarv 
concerns  of  the  PDR  are  the  software  design  and  the 
completion  of  requirements  set  forth  in  previously 
baselined  documents.  The  items  which  are  considered 
from  the  program  design  documents  include: 

a)  Computer  program  functional  flow  charts. 

b)  Storage  allocation  charts. 

c)  Control  functional  descriptions 
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d)  Structure  ar.d  org  ar.ization  of  the  faza  base. 

e)  Functional  interfaces.  [Bef.  33] 

Critical  Eesian  Reviews  -  These  reviews  are  ccniuctei 
as  individual'  system  programs  or  modules  are  speci¬ 
fied.  The  primary  concerns  here  are  that  Required 
standards  are  met,  all  prior  baselined  functions  are 
fulfilled,  and  that  prior  to  coding,  the  lowest  level 
of  design  detail  has  been  reached.  The  following 
items  [fief.  3$]  are  under  consideration  from  the 
program  specifications: 


a) 

Compat ability  of  design  with 
faces. 

functi 

or.al  inter- 

b) 

Data  base  interactions. 

c) 

Design  integrity  of  logic  diac 
storage  allocations,  and  flow  cf 

jrams, 

larts. 

algorithms. 

d) 

Hardware  interfaces. 

e) 

Human  interfaces. 

7.  Formal  Qualification  Review  -  This  is  conducted  upon 
completion  of  both  the  Functional  configuration  Audit 
ar.d  the  Physical  Configuration  Audit  and  after  all 
discrepancies  uncovered  by  these  audits  are 
corrected.  The  ourDose  here  is  to  ensure  that  the 
system  as  developed  and  tested  meets  all  requirements 
set  forth  in  the  Operational  a sauirements.  From  this 
review  it  is  determined  that: 

a)  User  requirements  have  been  satisfied. 

* 

b)  Documentation  is  sufficient  to  support  the  system 
throughout  its  life-cycle. 

c)  User  functions  are  adequately  described  and  docu¬ 
mented  . 

d)  Testing  is  sufficient  tc  ensure  user  confidence. 

e)  All  outstanding  deficiency  reports  have  been 

resolved. 

f)  The  outcome  of  the  review  will  either  be  a 
successful  completion  or  the  determination  that 
further  development  is  necessary  [Ref.  33]. 

8.  Post  Development  Review?  -  These  reviews  are 

conducted  as  the  operational  evaluations  of  the 

system  are  made.  The' focus  is  on  system  development 
problem  areas,  operational  difficulties,  and  unde¬ 
tected  deficiencies  [Ref.  33]. 

The  following  list  contains  the  major  audits 

conducted  by  the  NOSC  Software  Quality  Assurance  group 

during  the  software  development  process.  As  was  the  case 

with  the  reviews,.  a  brief  description  of  each  audit  is 

included. 
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1. 


Ir.-Proces s  Configuration  Audit  -  This  ssrves  as  -.he 
primary  means  or  validatma  that  development  or  a 
software  Configuration  Item  has  been  completed  satis¬ 
factorily.  Once  a  dooumeat  has  gone  thrcuah  the 
review  and  substantiation  process  (described  earlier) 
a  record  cf  the  review  forms  and  disposition  cf  defi¬ 
ciencies  is  maintained.  The  updated  document  and  all 
deficiency  reports  are  inputs  to  this  audit 
[Bef.  33]. 

2.  Functional  Configuration  Audit  -  Thj.s  is  a  critical 
comparison  of  an  item's  test/analysis  data  and  func¬ 
tional  specifications  to  validate  that  the  item  as 
designed  and  developed,  meets  all  the  functional 
performance  requirements  specified  in  its  development 
specification  fBef.  35]. 

3.  Physical  Configuration  Audit  -  This  is  a  comparison 
of  the  "as  built"  item  with  its  approved  and  released 
technical  documentation.  The  objective  is  to  ensure 
that  the  documentation  is  complete  and  is  appropriate 
for  operational  maintenance’  and  support' *  purposes 
[Bef.  35]. 

Both  the  Functional  Configuration  Audit  and  the 
Physical  Configuration  Audit  are  conducted  prior  to  the 
submission  cf  a  configuration  item  for  formal  acceptance. 
From  [Bef.  33]  the  purpose  of  these  audits  is: 

1.  Confirm  compliance  to  change  control  procedures  and 
tc  ensure  only  approved  changes  have  been  imple¬ 
mented. 

2.  To  ensure  that  objectives  are  sufficient. 

3.  To  ensure  the  develooed  software  produce  is  the  same 
as  the  specified  software  product. 


f.  Testing 


Software  test  and  evaluation,  as  conducted  by 
the  Software  Quality  Assurance  personnel,  is  designed  to 
ensure  that  the  software,  as  developed,  meets  the  original 
requirements  set  forth  by  the  user/sponsor  in  the 
Operational  Requirement  and  that  it  performs  as  defined  in 
the  system  documentation.  By  conducting  an  analysis,  tech¬ 
nical  evaluation,.  and  a  detailed  review  of  project 
documentation  the  necessary  inputs  to  prepare  test  plans, 
test  specifications,  and  test  procedures,  are  obtained. 
Th9se  documents  are  then  used  in  the  actual  testing  and 
evaluation  of  test  results  to  verify  that  the  system,  as 
developed  meets  all  technical  specifications  [Bef.  33]. 
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Following  is  a  list  of  the  various  rest  el  stents 
and  activities.  A  brief  discussion  of  each  is  included. 

1.  Test  Plan  -  This  is  written  by  Software  Quality 

Control  personnel  to  definr  the 

scope  of  tests  require^  to  assure  that  the  software 
meets  all  required  specifications.  Although  the  test 
plan  is  a  high  level  document,  from  which  the  test 
specifications  are  written,  it  still  must  identify 
the  degree  of  testing  and  the  specific  functions  to 
be  tested.  The  schedule  for  individual  tests  and  a 
summary  of  the  environment  to  be  used  are  also 
included  in  the  test  plan.  The  plan  is  reviewed  by 
the  software  developers  and  approved  by  the 
Prograa/Proj ect  Manager  [Ref.  36]. 

2.  Test  Specifications  -  Software  Quality  Control 

prepares  a  test  specification  for  each  test  m  the 
test  plan.  Based  on  requirements  set  forth  in  the 

desian  documentation  of  the  developing  system,  the 
test  specif ication  defines  the  basic  test  criteria 
and  the  general  methods  to  be  used  in  a  specific 
test.  Once  a  test  specification  is  prepared  it  forms 
the  basis  for  the  development  of  test  procedures.  In 
addition  to  defining  the  scope  of  the  specific  tests* 
test  specifications  state  the  purpose  or  the  test  ana 
identify  the  software,  hardware,  and/or  system  to  be 
tested.  There  must  be  sufficient  information  in  a 
test  specification  so  that  test  procedures  may  be 
developed  and  so  the  results  of  defined  tests  mav  be 
evaluated.  These  are  also  reviewed  bv  the  software 
developers  and  approved  by  the  Program/Project 
Manager  [Ref.  36]. 

3.  ‘  Test  Procedures  -  These  procedures  are  developed  bv 

the  Software  Quality  Control  oersonnel  using  test 
specifications,  users  manuals,  ar.d  other  relevant 
design  documentation.  The  prime  Durocse  of  the  test 

frocedures  is  to  present  detailed  instructions  for 
oth  test  execution  ar.d  the  evaluation  of  test 
results.  The  organization  and  structure  of  the 
processes  are  expressed  in  general  terms  in  the  test 
procedure  along  with  constraints  or  assumptions 
imposed  on  their  usage.  A  description  of  the  total 
equipment,  manpower,  computer  program,  and  supporting 
documentation  required  for  operation  is  also 

Provided.  All  hardware  or  software  revisions  or  modi- 
ications  must  be  specified  along  with  any  required 
pre-test  checkout  required  to  ensure  a  valid  test 
environment  [Ref.  35], 

«.  Software  Testing  -  Software  Quality  Control  personnel 
m$y  perform  testing  m  either  of  two  environments: 
simulated  or  "on-site".  In  a  simulated  environment 
certain  subsystems,  links,  and  peripherals  are  avai¬ 
lable  for  the  express  purpose  of  production  and 
testing.  "On-site"  testing  is  done  using  the  system 
in  real-life  environment.  The  policies  ana  procedures 
are  set  forth  in  the  test  plan,  specifications,  and 

froceduresused  in  accomplishing  software  testing 
Ref .  36  ]  . 

5.  Test  Reports  -  As  each  test  is  completed  a  Test 
Report  is  written  to  document  the  satisfactory  or 
unsatisfactory  completion  of  the  test.  Any  and  all 
deviations  from  tes*-  procedures  or  equipment  malfunc¬ 
tions  must  also  appear  on  the  Test  Report.  Each 
apparent  system  discrepancy  will  be  noted  by  the 
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submission  of  a  seperate  incident  report .  The  Test 
P.eoort  v:  11  reference  all  completed  rest  rr^c-iure 
seeps  to  allow  programmers  to' duDlicare  the  condi¬ 
tions  in  which  any  apparent  incident  was  discovered. 
All  completed  Test  Reports  become  a  parr  of  the 
permanent  system  documentation  [Hef.  36], 

g.  Problem  Reporting  and  Reviews 


As  was  the  case  with  TRW  the  area  of  change 
control  is  also  discussed  in  this  subsection.  The 
Configuration  Control  3oird  and  the  documents  used  in 
configuration  control  are  discussed  in  the  list  that 
follows. 


cen flour  a fit  n  Chang 

configuration  change 
changes  or  modifica 
items.  The  sponsor, 
ether  member  of  rhe 
changes  re  the  soft* 
lems  uncovered  dur 
presented  to  the  Con 
either  Engineering  C 
waivers.  The  CCS 
requests,  deviations 
mented  analysis, 
[Ref.  33].  The  CC 
from  all  project  af 
Manager  is  tae  cha 
representative  acts 
“■he  final  decision 
rests  with  the  chai 
from  prelect  par 
Development,  System 
Logistics  Suppor 

Facilities/Hardwara 


e  Control  -  The  ourocse  of 
control  is  to  manage  and' monitor 
tions  to  baselined  configuration 
user,  software  developer,  or  any 
Drolec*  oraar.ization  mav  oreDose 
are. '  These;  along  with  any  prob¬ 
ing  test  and  evaluation,  are 
figuration  Control  Board  (CCB)  as 
hange  Proposals,  deviations,  or 
in  turn  investigates  all  change 
,  and  waivers  and  based  on  docu- 
recommends  Droposed  action 
B  is  made  up  of  reDreser.tatives 
fected  activities.  The  Project 
irman  and  the  Quality  Assurance 
as  recording  secretary.  Although 
regarding  all  changes  ultimate!-/ 
rman,  hi  solicits  expert  advice 
tici pants  such  as  Software 
s  Engineering,  Quality  Control, 
t.  Fleet  User,  and 

management  [  Hef .  34], 


Request  for  Deviation  -  This  is  used  when  it  is 
necessary  to  depart  temporarily  from  documented 
requirements.  Dd  Form  1694  is  used  in  this  case 

[Ref.  34], 

Request  for  Waive;.  -  If.  an  item  fails  po  conform  to 
its  required  conng  uration  and  this  is  due  to  a 
development  error,  a  Request  for  Waiver  is  submitted. 
This  is  also  submitted  on  a  DD  Form  1634  [Ref.  34]. 

Engineering  Change  Orders  -  Once  approval  h*s  been 
granted.  Engineering  Change  Proposals,  Deviations, 
and  Waivers  are  implemented  through  an  Engineering 
Change  Order  [Ref.  34]. 

Specification  Change  Not;ce/Notice  of  Revision  -  Both 
these  documents  are  used  when  an  Engineering  Change 
Proposal  or  Waiver  affects  baselined  documents  or 
drawings.  DD  Form  1696  is  used  for  the  SCN  and  DD 
Form  1695  is  used  lor  the  NOR  [Ref.  34]. 
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Software  Trouble/Incident  Report  -  This  is  used  for 
reporting  all  deviations  iron  t=st  Procedures,  equip¬ 
ment  naif  unctions  and  software  anamclies  [8ef.  36j. 
These  become  a  part  of  the  software  Test  Report  and  a 
permanent  part  of  system  documentation. 

h.  Program  Library  Control 

Program  Library  Control  is  designed  to  maintain 
and  control  a  systems  computer  programs  and  all  changes  to 
these  programs.  Software  Quality  Assurance  personnel  are 
responsible  for  approving  all  changes  to  library  programs. 
This  system  not  only  provides  baseline,  error-free,  patch 
free  programs  for  test  and  evaluation  but  it  also  insures 
that  all  approved  changes  are  incorporated  into  the  programs 
[Ref.  37}.  TRW  uses  a  system  similar  to  this. 

i.  Benefits 

In  [Ref.  33]  it  states  that  NOSC  has  realized 
the  following  benefits  through  the  establishment  of  its 
Software  Quality  Assurance  program: 

1.  It  establishes  a  controllable  structure  in  the  soft¬ 
ware  development  process. 

2.  It  assures  certain  elements  of  quality  in  every  chase 
of  the  development. 

3.  By  reducing  rework  substantially,  it  provides  for  th° 
satisfaction  of  requirements. 

4.  The  substantia],  seduction  in  the  apcuni  of  rework  has 
lead  to  a  significant  savings  in  life-cycle  costs. 

3.  Gener al  Electric  Company 
a.  Introduction 

Tha  software  quality  assurance  activities  at  two 
separate  divisions  of  the  General  Electric  Company  will  be 
discussed.  One  discussion  will  cover  the  Electronic  Systems 
Division  located  in  Syracuse,  New  York.  The  other  will  cover 
the  Space  Division  located  in  King  of  Prussia,  Pennsylvania. 
Neither  discussion  will  be  extensive  as  both  divisions  use 
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quality  assurance  techniques  which  ire  similar  to  the  -wo 
organizations  previously  described. 

b.  Electronic  Systems  Division 

(1)  .  Quality  Assurance  Objectives.  John 
McKissick  Jr.  and  Robert  A.  Price  point  out  in  [Ref.  38]  the 
objective  of  the  Computer  Software  Quality  Assurance  (CSQA) 
Program  at  the  G.  E.  Electronic  Systems  Division  is  to 
ensure  that  software  delivered  under  a  contract  meets  the 
requirements  of  the  contract.  As  can  be  seen  this  is  similar 
to  the  organizations  previously  discussed. 

(2)  .  Quality  Assurance  Planning.  I  he  manager 
of  Computer  Software  Reliability  and  Quality  Assurance  and 
dedicated  technical  specialists  are  responsible  for  planning 
ana  implementing  the  QA  Program.  In  addition  to  providing 
staff  support  to  project  managers  the  manager  of  Computer 
Software  Reliability  and  Quality  Assurance  reports  directly 
to  the  manager  of  Reliability  and  Quality  Assurance.  I- 
should  be  noted  tha*  the  Computer  Sof-ware  Reliability  and 
Quality  Assurance  group  is  organizationally  independent  of 
Software  Engineering. 

After  reviewing  both  the  Computer  Program 
Management  Plan  and  the  Software  Standards  and  Procedures 
Manual,  Quality  Assurance  personnel  develop  the  Computer 
Software  Quality  Assurance  Plan  [Ref.  38].  The  plan  defines 
all  activities  which  control  and  assure  computer  software 
quality.  The  plan  is  developed  using  Mii-s-52779A,  and  it 
identifies  the  organizational  component  responsible  for  each 
activity. 

(3)  •  Software  5iao,darJs.  Similar  to  TRW  the  G. 
E.  Electronic  Systems  Division  also  develops  a  Software 
Standards  and  Procedures  Manual.  This  document,  which  is 
primarily  used  by  the  programming  teams,  establishes  rules, 
guidelines,  and  limitations  which  are  to  be  observed  in 
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generating  software  designs  and  cola  which  will  have  the 
properties  of:  consistency,  readability,  and  quality 

[Ref.  38].  The  structure  and  content  of  the  Software 
Development  Notebook  (SDN)  is  also  defined. 

The  SDN  is  similar  to  the  Unit  Development 
Folder  used  by  TRW.  It  is  a  simple  loose-leaf  notebook  which 
is  established  during  the  Preliminary  Design  phase  for  each 
Computer  Program  Component  (module) .  It  provides  a  common 
collection  point  for  all  information  dealing  with  a  CPC  and 
it  is  maintained  and  updated  throughout  the  remaining  phases 
of  the  software  development  [Ref.  38]. 

The  SDN  is  broken  down  i nr o  the  following 
sections:  Requirements,  Detail  Design,  Functional 

Capabilities,  Code,  Test  Case  Descriptions,  Test  Case 
Results,  Software  Problem  Reports,  ar.a  miscellaneous 
Information.  With  the  exception  of  Software  Problem  Reports 
and  Miscellaneous  Information,  each  section  contains  a  cover 
sheet  shewing  schedule  dates,  actual  completion  dates,  and 
review  and  approval  signatures  [Ref.  38]. 

The  SDN,  like  TRW*  s  DDF ,  serves  as  the 
principle  working  document  of  the  programmer.  In  addition  it 
provides  management,  the  customer,  and  QA  personnel  visi¬ 
bility  into  the  design,  status,  and  quality  of  the  software 
under  development  [Ref.  38]. 

SDNs  are  audited  monthly.  This  audit  may 
be  on  either  an  announced  or  unannounced  basis. 

(4)  .  quality  Assu ranee  Reviews  and  Audits. 
Audits  and  reviews  similar  to  those  conducted  by  both  TRW 
and  NOSC  are  conducted  at  the  G.  E.  Electronic  Systems 
Division.  Internal  reviews  are  conducted  by  software  and 
systems  engineers  who  have  not  contributed  to  the  design 
under  review.  The  review  chairman  is  responsible  for  imple¬ 
menting  the  review  plan  which  was  approved  by  Quality 
Assurance  and  the  Project  Manager. 
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After  the  internal 


reviews  are  ccr.duc 


joint  customer/c  cnxractor  reviews  are  held.  These  reviews 
provide  a  technical  forum  for  better  mutual  understanding  of 
the  performance  requirements  allocated  to  the  computer  soft¬ 
ware,  and  of  the  design  approach  selected  to  meet  these 
requirements  [Ref.  38% 

(5)  .  Testing.  At  the  G.  E.  Electronic  Systems 
Division  the  Test  Plan  is  developed  during  the  preliminary 
design  phase  of  the  development  process.  Test  procedures 

are  developed  during  the  detailed  design  phase.  Actual 
testing  occurs  in  the  final  four  phases  of  the  development 
process:  Cede,  Debug  and  Dr.it  Test,  Development  Testing, 
Integration  Testing,  and  Acceptance  Tasting  [Bef.  38], 

Dnit  testing  is  conducted  on  individual 
routines  to  reveal  coding  errors,  computational  errors, 
improper  input  handling,  inappropriate  error  messages,  and 
incorrect  formatting  and  content  of  output.  Development 
testing  takes  the  prviousiy  tested  routines  and  combines 
them  to  form  Computer  Program  Components  (CPC)  .  The  func¬ 
tional  capabilities  of  the  CPC  is  are  then  verified. 
Integration  testing,  which  is  performed  by  an  independent 
test  team,  combines  CPCs  and  verifies  the  correct  sequencing 
of  components,  compatible  component  interfaces,  and  proper 
data  routing.  Acceptance  testing,  which  is  also  done  by  an 
independent  team,  verifies  system  level  functional  require¬ 
ments.  These  include  overall  timing  and  the  ability  to 
handle  the  total  input  load.  As  at  TRW  and  NOSC  QA  personnel 
witness  all  testing  [Bef.  3  8% 

(6)  .  Problem  R e porting  and  Review.  Like  both 
TRW  and  NOSC,  G.  E.  Electronic  Systems  Division  has  a 
Configuration  Control  Board  which  reviews  and  approves  all 
changes  to  the  software.  As  in  both  the  previously  described 
organizations.  Quality  Assurance  personnel  are  members  of 
this  board.  Quality  Assurance  personnel  are  also  responsible 
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for  verifying  thrt  all  approved  changes  hav*  been  incorpo¬ 
rated.  This  is  similar  to  both  MOST  and  TRW. 

A  Software  Problem  Report  (SPR)  defines 
and  documents  a  problem  and  the  test  conditions  under  which 
it  occured.  Quality  Assurance  receives  a  copy  of  all  SPRs. 
In  addition  they  maintain  a  listing  of  all  outstanding  SPRs 
and  the  party  responsible  for  corrective  action.  Once  a 
problem  has  been  corrected  Quality  Assurance  annotates  its 
copy  of  the  SPR  with  the  way  in  which  the  problem  was 
corrected  [Ref.  38]. 

All  coding  changes  are  authorized  using  a 
Software  Change  Order  (SCO)  .  Changes  to  hardware  and  soft¬ 
ware  specifications  are  documented  on  a  Specification  Change 
Notice  (SCN)  [Ref.  38]. 

(7)  .  Benefits.  An  effective  Quality  Assurance 
Program  allows  the  G.  E.  Electronic  Systems  Division  to 
deliver  computer  software  which  meets  all  contractual 
requirements.  In  addition  it  provides  management  visibility 
into  the  software  development  process. 

c.  Space  Division 

(1)  .  Quality  Assurance  Objectives.  The  Quality 
Assurance  Prograu  which  was  reviewed  has  been  in  effect  at 
the  General  Electric  Company*s  Space  Division  sine  e  1978. 
The  primary  objective  of  the  program  is  to  ensure  that 
delivered  software  meets  all  contractual  requirements.  A 
secondary  objective,  which  is  designed  to  help  secure 
contracts,  is  to  define  and  implement  specific  measures 
designed  to  ensure  delivered  software  incorporates  the 
features  necessary  to  achieve  testability,  maintainability, 
reliability,  etc.  [Ref.  29]. 

(2)  .  Assu£att£§.  Planning.  As  is  the 

case  in  the  three  previously  described  organizations  the 
primary  job  of  the  Quality  Assurance  group  is  to  prepare  the 
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QA  Plan.  Once  the  QA  Plan  has  been  developed  Quality 
Assurance  turns  its  attention  to  the  implementation  and 
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(3)  .  Software  Standards.  Each  programmer 
working  on  a  given  project  receives  a  copy  of  the 
Programming  Standards  Document  (PSD)  ,  which  outlines  the 
standards  to  be  used  to  produce  a  high-quality  software 
product.  It  is  not  enough  however  that  the  programmer  be 
given  the  PSD  and  left  to  go  along  his  merry  way.  Training 
sessions,  conducted  by  the  Software  Development  group,  are 
held  to  explain  the  content  of  the  PSD.  It  is  the  job  of  the 
Quality  Assurance  group  no  verify  that  each  programmer 
participates  in  the  training  sessions  [Hef.  29]. 

(4)  .  Quality  Assurance  5 sv lews  and  Audits.  The 
G.  E.  Space  Division  conducts  reviews  and  audits  similar  to 
those  of  the  previous  three  organizations.  They  do,  however, 
give  special  attention  to  the  developmentof  software  inter¬ 
faces.  The  development  of  the  software  interfaces  is  done 
by  the  Software  Development  Group.  However  the  System 
Engineering  group  is  responsible  for  developing  the 
Interface  Contrcl  Documents  (ICDl  .  The  purpose  of  this 
process  is  to  provide  a  timely  and  complete  definition  of 
interface  details  and  to  provide  a  continuous  in-depth 
review  process  [  Hef .  29]. 

(5)  .  Testing.  The  Quality  Assurance  role  in 
the  actual  testing  is  minor.  The  bulk  of  the  Quality 
Assurance  groups  work  is  done  prior  to  actual  testing. 

Quality  Assurance  first  identifies  exactly 
what  is  to  be  tested  and  at  the  same  time  develops  a 
detailed  definition  of  the  test  environment.  They  then 
explicitly  define  both  the  test  and  the  evaluation  process, 
especially  success/f ailura  criteria. 


During  the  actual  resting,  Quality 
Assurance  acts  as  a  monitor  to  ensure  that  the  previously 
defined  procedures  are  being  followed.  They  also  ensure  that 
any  changes  are  documented  correctly  [Ref-  29]. 

(6)  .  Problem  R  epor  ting  and  Review.  Any  prob¬ 
lems  uncovered  during  the  testing  process  are  documented 
using  a  Discrepancy  Report  (DR)  .  3r.ce  testing  is  complete  a 
post-test  meeting  is  held  and  the  Discrepancy  Reports  are 
assigned  to  individuals  for  resolution.  The  Software  Quality 
Assurance  group  monitors  all  outstanding  DRs  to  ensure  that 
all  problems  are  corrected. 

The  Discrepancy  Reports  serve  as  a  measure 
of  product  quality.  The  software  2A  group  analyzes  •‘•he  data 
provided  by  the  DR  and  prepares  a  statistical  report.  In 
[Ref.  29]  Stephen  L.  Stamm,  Manager  of  Productivity  Programs 
at  the  G.  E.  Space  Division,  points  out  that  such  things  as 
the  number  of  DRs,  the  frequency  distribution  cf  DRs  by 
type, .the  mean  time  of  closure  (correction),  and  the  DR  rate 
as  a  function  of  product  1  if e  can  be  used  by  management  to 
identify  weak  spots  in  the  software  implementation  process. 

(7)  .  Program  L  ibrary .  A  system  similar  to  that 
at  NO SC  is  used  at  the  G.  E.  Space  Division. 

(8)  .  Oualit v  Assurance  Tools.  Automated  soft¬ 
ware  cede  analysis  tools  are  used  as  part  of  the  test 
program  to  uncover  code  in  the  final  product  which  has  never 
been  executed.  If  this  is  the  case  it  is  determined  if  there 
is  a  hole  in  the  test  program,  a  possible  flaw  in  the 
product  design  or  just  some  superfluous  code  in  the  finished 
product  [Ref.  29]. 

(9)  •  Lessons  The  following  elements 
[Ref.  29]  have  been  found  by  the  G.  E.  Space  Division  to  be 
necessary  for  a  successful  Software  Quality  Assurance 
Program: 

1.  The  Software  .Quality  Assurance  Plan  must  have  high 
project  visibility,  it  must  define  the  SQA  program  at 


a  level  of  detail  sufficient  to  allow  implementation, 
ar.d  it  must  have  the  project  ar.i  company  mana^'^ents 

whole-hearted  support. 

2.  The  application  of  special  software  engineering  tech¬ 
niques  by  the  progra  mining  staff  specifically  targeted 
at  increasing  product  quality. 

3.  The  project  organization  must  distribute  the  Software 
Quality  Assurance  Program  responsibility,  placing  SQA 
tasks  where  the  capability  really  exists. 

4.  The  ability  to  measure  the  effectiveness  of  the  QA 
program  ana,  if  possible,  the  quality  of  the  end 
product. 

5.  Software  Quality  Assurance  personnelaust  be  an 
integral  part  of  the  project  team. 


s.  conclusion 

Four  separate  Software  Quality  Assurance  organizations 
have  been  discussed  in  this  chapter.  However  they  are 
similar  in  several  ways. 

First  the  Software  Quality  Assurance  groups  at  each  of 
these  organizations  become  involved  with  the  project  early 
in  its  life-cycle.  Each  of  The  Quality  Assurance  groups  is 
involved  in  every  phase  of  the  development  process.  Each  of 
these  organizations  recognizes  the  fact  that  an  effective 
Quality  Assurance  organization  allows  them  to  deliver  a 
quality  software  product  which  meets  all  contractual 
requirements.  They  are  also  in  agreement  on  the  fact 
Software  Quality  Assurance  saves  money  in  the  development 
process  by  identifying  and  correcting  errors  early  in  the 
process.  They  also  agree  that  an  effective  QA  program 
provides  both  management  (Corporate  and  Project)  and  the 
customer  wi^h  visibility  into  the  development  process.  The 
view  of  software  as  a  product  is  also  a  similarity  among 
these  organizations.  Finally  all  these  groups  agree  that 
Software  Quality  Assurance  groups  do  not  create  quality  in  a 
project  but  tha*  it  is  in  fact  part  of  the  job  of  every 
project  member. 
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IV.  JHB  fiOALITY  APPROACH  AT  FMSO 

A.  STROCTORB 

Within  the  Naval  Material  Command,  the  Fleet  Material 
Support  Office  (FBSO)  is  a  field  activity  sponsored  by  the 
Naval  Supply  Systems  Command.  FMSD  performs  two  major  func¬ 
tions  in  fleet  logistical  support.  The  one  discussed  here 
is  that  cf  principal  Navy  Central  Design  Agency  (CD A)  for 
automated  supply,  financial,  maintenance,  and  logistical 
systems,  a  process  which  consumes  the  majority  of  FMSO 
resources. 

A  general  description  of  the  organizational  elements 
directly  related  to  the  central  systems  development  process 
is  depicted  in  figure  4.  1  Within  the  organization  the 
Comptroller  Department  (Code  91)  and  Management  Department 
(Code  92)  are  the  two  departments  that  can  be  considered  as 
staff.  The  other  six  CDA  departments  are  production 
oriented  or  support  the  production  effort  and  are  considered 
as  line  organizations  which  are  directly  responsible  for  the 
development  and  maintenance  of  standard  Automated  Data 
Processing  (ADP)  systems  [Ref.  39]. 

A  system  is  considered  to  be  an  organized  set  of  ADP 
hardware,  envir onmental/app Ideation  software,  and  documented 
procedures  designed  to  automate  the  basic  management  and 
operating  processes  for  a  customer  sits  or  group  of  customer 
sites  with  common  mission  responsibilities.  "Documented 
procedures"  as  used  above  refers  to  the  applicable  ADP 
related  and  non-ADP  related  procedures  established  to 
support  the  hardware  and  software  aspects  of  the  system 
(Ref.  39]. 
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MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BURLAU  Of  STANDARDS-  196J-A 


Figure  4.1  Organizational  Structure 


The  primary  role  of  the  Management  Department  is  to 
coordinate  with  and  support  the  efforts  of  the  CDA 
Production  Departments.  In  this  capacity  the  branch  that  is 
of  most  relevance  to  this  paper  is  that  of  the  Quality 
Control  Branch.  The  CD A  Development  Process  Model,  figure 
4.2,  reflects  all  of  the  basic  steps  appropriate  to  ensuring 
that  each  CDA  tasking  received  by  FHSO  is  effectively 
managed  and  results  in  a  high  quality  product  being  released 
for  use  by  the  customer.  The  model  covers  all  projects, 
large  and  small,  new  developments  or  maintenance.  However, 
it  is  anticipated  that  some  of  the  steps  in  the  model  may 
not  be  applicable  to  all  projects.  Therefore,  an  explicit 
decision  by  the  appropriate  level  of  management  is  required 
in  order  to  exclude  process  steps  determined  not  applicable 
on  a  project.  As  the  model  is  followed  through,  note 
specifically  the  areas  that  deal  with  symbols  equating  to  92 
QC  and  92  QC  Optional  (Hef.  40]. 


B.  THE  QUALITY  IHOCESS 

As  a  new  requirement  is  received  by  FMSO  a  aechanisa  is 
activated  to  ensure  thar  the  output  produced  will  meet  the 
user's  expectations.  This  mechanist  at  FMSO  is  called  the 
quality  process,  that  is,  an  attitude  that  extends  from  the 
individual  programmer  all  the  way  through  the  systems  devel¬ 
opment  cycle  up  to  top  aanagement.  By  design,  it  is  a 
multi-layered  approach  to  achieving  quality  that  begins  with 
the  System  Development  Quality  Process  (SDQP) .  Falling 
under  the  SDQP  are  all  -he  separate  requirements  that  will 
eventually  produce  a  workinc  ADS.  At  each  stage  of  develop¬ 
ment,  quality  standards  are  imposed  upon  all  personnel  at 
all  levels  within  the  chain  of  comaand  beginning  with  the 
feasibility  study  and  requirement  definition,  proceeding 
through  the  functional  design,  coapurer  design,  program 
development,  testing,  operations  and  maintenance,  and  ending 
with  the  Program  Trouble  Beports  (PTR)  .  Within  each  of 
these  particular  evolutions  labeled  above  are  subsections 
that  must  be  completed  before  the  process  evolves  further. 

Layered  on  top  of  the  SDQP  are  the  Quality  Control 
Mechanisms.  These  Quality  Control  Mechanisms  provide  the 
ability  to  ensure  that  a  quality  and  error  free  product  is 
in  fact  produced.  The  methods  utilized  to  accomplish  this 
are  good  project  management  during  the  requirement  defini¬ 
tion  and  functional  design  stage,  sound  data  base  management 
during  functional  design  and  computer  design,  an  effective 
verification  process  during  computer  design  and  program 
development,  _  proper  validation  procedures  during  program 
development  and  testing,  and  a  satisfactory  prototype/op 
review  during  testing  and  operations/  maintenance  of  a 
system. 
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On  top  of  both  the  SDQP  and  Quality  Control  Mechanisms 
we  have  the  Quality  Production  Concepts.  These  include 
structured  processing,  standard  data  element  usage,  uniform 
standards,  methods  and  procedures,  improved  programming 
techniques  and  training.  When  all  of  these  various  layers 
are  utilized  and  iapleaemted  as  a  whole  we  have  the  philo¬ 
sophy  of  PMSO  towards  producing  a  quality  product. 

C.  QUALITY  ASSURANCE  VS.  QUALITY  CONTROL 

Quality  Assurance,  as  defined  by  PMSO  [Ref.  40],  is  a 
line  management  responsibility.  As  such.  Line  Supervisors 
are  accountable  for  enforcing  the  application  of  standard 
procedures  that  have  been  developed  for  the  primary  purpose 
of  insuring  accuracy,  thoroughness  of  method,  simplici-y  in 
design,  adequacy  of  testing  and  clarity  of  documentation  of 
ADS  development.  To  aid  all  levels  of  personnel  within  the 
various  line  departments  in  the  accomplishment  of  their  own 
specific  requirements  for  quality,  guidelines  have  beer- 
developed  that  include  NAVSUP  PUBs  506,  507,  508,  CD  A 
DEVELOPMENT  HANDBOOK,  CD A  MANAGEMENT  HANDBOOK,  PMSO  Internal 
Instructions,  and  various  other  documented  and  undocumented 
departmental  procedures.  Everything  written  and  documented 
must  be  in  compliance  with  these  standards.  It  is  the  indi¬ 
vidual  person's  responsibility  to  ensure  that  they  are  in 
compliance  with  these  stahdards. 

Quality  Control,  as  defined  by  PMSO  [Ref.  40],  is  the 
responsibility  of  the  Management  Department  and  specifically 
the  Quality  Control  Branch.  Quality  control  procedures  are 
those  actions  that  are  taken  as  an  ADS  is  being  developed  to 
insure  that  all  the  required  QUALITY  ASSURANCE  procedures  , 
those  actions  performed  by  the  line  department  personnel, 
will  be  complied  with  to  produce  a  reliable  and  error  free 
ADS  product.  Quality  Control  than  is  primarily  a  review 
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function  to  be  performed  by  the  Quality  Control  Branch.  Ir. 
this  area  they  are  responsible  for  ensuring  chat 
designated/high  interest  ADS  projects  conform  with  standards 
of  completeness,  accuracy,  clarity,  and  all  other  applicable 
quality  standards  that  applied  to  the  line  quality  assurance 
program  have  been  achieved. 


D.  SPECIFIC  QU  A  LIT  T  CONTROL  RESPONSIBILITIES 


As  quality  control  is  a  review  function  and  given  the 
limited  number  cf  personnel  assigned  to  the  branch,  not 
every  output  that  is  provided  by  FMSD  will  be  reviewed  by 
the  Quality  control  Branch.  The  Quality  Control  Branch 
will,  however,  be  directly  involved  with  those  projects 
designated  high  priority  or  of  special  interest  to  the 
Command.  All  other  projects  will  be  reviewed  on  a  priority 
basis  as  the  manhours  that  can  be  devoted  to  the  futher 
enhancement  of  the  quality  effort  become  available.  Their 
area  of  responsibility  is  enormous  and  their  tasks  numerous. 
Therefore,  only  areas  of  major  responsibilities  are  listed 
below  fRef.  40], 

1.  Review  of  Functional  Descriptions  for  compliance  with 
standards . 

2.  Participate  in  a  system  design  review  to  ensure  that 
the  design  has  considered  all  of  the  proposed  system 
requ  irements. 

3.  Review  of  System  Specifications  for  compliance  with 
standards . 

4.  Review  of  Program  Specifications. 

5.  Review  of  the  Main^e nance  Manual,  Operations  Manual, 
and  all  other  applicable  manuals. 


6. 

7. 


Prepare  an  analysis  of  PT?s  received  as  to  gause  or 
symptom  and  recommend  possible  corrective  action. 


The  Quality  Control  Branch  will  selectively  reyaew 
test  plans  ana  tests  for  compliance  with  Quality 
Assurance  guidelines.  In  the  performance  of  this 
task,  they  may  desk  check  all  applicable  data  or  they 
may  elect  to  attend  the  review  conference  held 
between  the  programmer  and  an 
the  results  or  the  test. 


task,  they  may  desk  check  all  applicable  data  or  they 
may  elect  to  attend  the  review  conference  held 
between  the  programmer  and  analyst  as  they  discuss 
the  results  or  the  test. 


8.  Review  the  I  spies ent ation  Plan. 
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Quality  Control  is  responsible  for  post  implementa¬ 
tion  visits  to  selected  sites  cf  iese.gr.ared '  pr  o-'-otr 
to  determine  whether  the  product  released  is  working 
satisfactorily,  meets  the  needs  of  the  user,  and  is 
being  utilized  correctly. 

10.  Perform  a  Quality  Assurance  Review  of  all  designated 
ADS  programs. 

E.  TESTING  TO  ENSURE  QUALITY 

The  testing  process  involves  many  different  personnel 
and  includes  man y  different  responsibility  assignments.  For 
example,  the  individual  programmer  assigned  to  the  project 
is  responsible  fcr  reasonable  testing  for  all  error  condi¬ 
tions  that  could  occur  in  the  program  and  for  providing 
support  for  the  systems  test  required  for  the  application. 
The  Lead  programmer  is  responsible  for  developing  the  *est 
plan  for  system  testing  and/or  string  testing. 

The  Systems  Analyst  is  responsible  for  assisting  the 
Lead  Programmer  in  planning  and  coordinating  the  srring 
testing/system  testing  to  determine  that  all  the  programs  in 
the  application  produce  the  required  output  when  run  in 
total.  The  Systems  Analyst  has  a  primary  responsibility  of 
approving  and/or  selecting  the  test  data  used  for  systems 
testing. 

The  Systems  Designer  is  to  participate  in  reviews  of 
test  output  to  insure  that  testing  of  the  ADP  program  has 
been  adaquete.  After  the  processes  above  are  completed,  the 
Quality  Control  Branch  will  selectively  review  test  plans 
and  test  results  for  compliance  with  all  Quality  Assurance 
Guidelines. 

1.  HEgs  Q1 

During  the  testing  cycle  there  are  primarily  four 
different  methods  utilized  to  check  the  programs. 

1.  Unit  Test  -  A  single  program  that  is  checked  by  the 
programmer  responsible  for  al30  writting  the  code.  A 


Or.it  Test  Review  is  scheduled  between  the  analyst  and 
programme z  with  zhe  test  results  being  reviewed  to 
ascertain  if  futher  testing  is  warranted. 

2.  String  Test  -  Each  program  release  which  requires  the 
execution  of  other  progress  in  actual  production  will 
be  string  tested.  Programmers  are  responsible  for 
writing  the  test  plan  to  ensure  that  all  major  paths 
and  functions  that  will  utilize  the  new  program  are 
checked. 

3.  System  Test  -  Each  new  ADS  or  major  change  involving 
more  than  one  application/operation  or  package  in  an 
existing  ADS  will  ba  subjects!  to  a  system  test  that 
will  evaluate  the  specific  system  as  a  whole.  The 
overall  responsibility  for  the  test  will  be  giver,  to 
the  Lead  Programmer.  The  Quality  Control  Branch  will 
evaluate  the  results  to  ensure  that  the  system  meets 
design  objectives. 

4.  Integrated  Sysxems  Test  -  This  test  will  be  designed 
to  test  all  program  interfaces,  database  interfaces, 
and  all  internal  and  external  applications  for 
correctness  of  data  flow.  The  Lead  Programmer  and 
Lead  CDA  Department  are  responsible  for  preparing  a 
formal  test  plan  and  conducting  the  test.  The  Quality 
Control  Branch  may  or  may  not  be  assigned  as  an 
overall  monitor  for  this  procedure  but  will  be 
required  to  review  the  results. 

F.  SYSTEM  RELEASE  PHOCBDORES 

Upon  completion  of  the  required  evaluations  and  for 
specified  projects,  the  Line  Departments  involved  will 
forward  to  the  Quality  Control  Branch  all  applicable  docu¬ 
mentation  for  review.  After  this  review  has  been  completed 
the  complete  package  is  sent  back  to  the  CDA  Department  Line 
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Manager.  when  the  Manager  is  satisfied  that  the  prcaraa 
meets  the  requirements  and  that  all  quality  assurance  stan¬ 
dards  have  been  met,  the  Line  Manager  will,  by  his  signature 
release  the  program  for  use.  This  will  then  terminate  the 
Systems  Development  Process. 

6.  ETALUATIHG  THE  QUALITY  PROGRAM 

It  is  important  to  realize  that  at  FMSO,  Quality  Control 
is  not  intimately  concerned  with  the  daily  operations  of  the 
line  departments,  but  is  concerned  with  assessing  the 
results  of  the  line  departments.  aith  this  perspective  in 
mind,  the  design  and  execution  of  rha  Quality  Assurance  Plan 
is  considered  to  be  an  integral  part  of  -he  production 
process  itself.  Quality  assurance  for  software  consists  of 
the  formal  application  of  standards  and  execution  of 
required  tests,  and  then  the  assessment  of  results. 
Enforcement  of  the  standards  and  the  necessary  adjustments 
of  the  process  is  one  of  the  responsibilities  of  the  Quality 
Control  Branch. 

Although  the  Quality  C  cr.tr ol  3ranch  is  deeply  embedded 
within  the  Management  Department,  they  have  performed  in  an 
extremely  professional  manner.  When  given  an  adequate 
amount  of  time  and  an  opportunity  to  perform  in  their 
primary  role  as  reviewers,  they  have  always  met  the  chal¬ 
lenge.  But  this  opportunity  to  excel  does  not  always  occur 
as  it  should,  and  FMSO  is  no  different  than  any  other  soft¬ 
ware  producing  organization.  As  the  project  completion  date 
draws  near,  usually  the  first  item  to  be  called  excessive  to 
the  program  is  quality  control.  At  FMSO,  the  project 
manager  is  responsible  for  ensuring  that  enough  time  is 
alloted  during  the  SDP  so  that  Quality  Control  has  a  chance 
to  evaluate  the  program.  This  astimate  of  the  amount  of 
time  it  will  take  to  complete  a  project  is  a  most  difficult 
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one,  but  dates  have  to  be  set  and,  on  occasion,  met.  The 
authors  are  unable  to  cite  specific  examples  of  Quality 
Control  being  cut  short,  but  when  the  amount  of  work  that  is 
to  be  accomplished  and  the  number  of  personnel  assigned  to 
accomplish  it  are  compared,  the  assumption  can  be  made  that 
it  has  occurred  ,  either  as  a  result  of  internally  or  exter¬ 
nally  generated  pressure  to  comply  with  a  due  date. 

When  the  value  of  a  product  is  very  high,  such  as  on 
command  designated  /special  interest  projects,  each  item 
produced  is  individually  inspected  and  gone  over  in  fine 
detail  at  all  levels,  from  the  programmer  through  all  the 
reviews,  and  finally  top  management.  However,  under  less 
important  conditions,  the  selection  and  review  process  of 
products  is  not  as  critical,  and  for  very  minor  projects  it 
need  not  be.  This  does  not  mean  that  the  product  is  any 
less  important  to  the  user  but  merely  implies  that  all 
projects  other  than  the  exceptionally  simple  ones  should 
also  receive  the  same  degree  of  scrutiny  that  special 
projects  do. 

The  reality  of  the  present  situation  dictates  that  the 
above  process  is  not  feasible  at  this  time.  With  such  a 
small  staff  in  the  Quality  Control  Branch  and  the  multitude 
of  tasks  assigned,  it  is  physically  impossible  to  meet  all 
of  their  requirements,  and  priorities  must  be  established. 
With  an  ongoing  review  of  the  number  of  Computer  Specialists 
that  can  be  justified  within  the  Quality  Control  Branch,  it 
is  essential  that  the  justification  be  met  and  billet 
descriptions  constructed  to  place  more  people,  not  less, in 
this  tost  important  branch  if  FMSO  13  to  continue  with  its 
present  emphasis  upon  producing  a  quality  product. 

The  authors  feel  that  the  Quality  Control/Quality 
Assurance  program  at  FMSO  is  highly  competitive  with 
similiar  organizations.  Their  thorough  and  most  agressive 
instructions  and  standards  are  excellent,  and  if  the  present 
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level  of  enforcement  is  continued  will  greatly  enhance 


product  serving  the  Fleet.  Quality  Control/  Quality 

Assurance  does  happen  at  F»lSO,  primarily  due  to  the  struc¬ 
tured  process  and  to  the  professionals  that  work  and  manage 
the  organization.  The  degree  to  which  it  happens  is  an 
evolving  entity. 

At  the  present,  time  the  only  effective  documented  method 
to  measure  the  application  of  quality  practices  within  FMSO 
is  the  analysis  and  evaluation  of  the  Program  Trouble 
Reports  (PTR)  .  ETRs  may  be  submitted  during  any  step  of  the 
systems  development  process  or  after  the  program  has  been 
sent  to  the  field  user.  As  a  PTR  i?  received  at  FWSO  it  is 
routed  to  the  Project  Control  Branch-  where  is  is  logged  in 
and  then  sent  to  the  department  that  issued  the  program  with 
which  the  PTR  is  concerned.  The  department  involved  then 
decides  if  the  ETR  is  of  a  crixical  or  non-crixical  nature 
and  then  proceeds  to  work  on  it.  PTRs  are  classified  in 
two  ways,  critical,  which  has  a  significant  impact  upon 
daily  routine,  and-  non-crit  ical ,  which  has  less  of  an  impact 
and  can  temporarily  be  delayed.  Critical  PTRs  will  be 
corrected  as  socn  as  possible  and  normally  within  threa 
working  days  from  receipt  of  sufficient  information  to  allow 
the  CDA  to  act.  Non-critical  PTRs  will  be  corrected  as  soon 
as  possible  after  receipt  of  sufficient  data  that  will  allow 
the  CDA  to  act. 

The  Quality  Control  Branch  performs  a  quarterly  analysis 
of  all  PTRs  received  with  special  emphasis  on  the  most 
common  type  of  problem,  which  steps  of  the  systems  develop¬ 
ment  process  create  the  most  errors,  identification  of 
trends,  and  if  possible,  recommendations  for  corrective 
actions. 

A  survey  of  the  PTR  Analysis  Report  foe  the  Second 
Quarter  CT  82  reveals  the  following:  [Ref.  41], 
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.  A  total  of  385  PTRs,  of  which  275  were  complete,  92 

W61*®  C^^O’^X3^/  ^  P  **■  r2  ^  ^  'T'  U  ^  r?  <5  ^  ^  t  ^  1 

were  due  to  either  being  invalid  or  already  in  exis¬ 
tence.  A  reclassified  PTR  was  one  that,  when 
submitted,  it  was  felt  by  the  user  that  the  program 
was  not  performing  as  desirsd  or  requested  but  upon 
researching  the  problem  it  was  found  that  all 
requirements  had  been  met.  To  meet  the  new  require¬ 
ment  of  the  user  a  new  project  would  have  to  be 
designated. 

2.  Of  the  covpleted  PTRs,  38  were  critical  and  237  r.on- 
crit  leal . 

3.  The  FMSO  average  number  of  manhours  to  fix  a  PTR  was 
16  for  a  critical  PTR  and  21  for  a  nor.-critical  PTR. 
One  possible  reason  for  the  difference  in  times  is 
the  experience  level  of  personnel  assigned  to  repair 
a  program.  A  more  experienced  programmer  is  gener¬ 
ally  assigned  to  a  critical  PTR. 

а.  The  average  number  of  days  for  FflSO  to  complete  a 

critical  PTR  is  15.5  and  151.6  to  complete  a  ton- 
critical  FTR .  This  figure  may  be  skewed  toward  the 

high  side  very  easily  because  of  the  small  number  of 
critical  FTRs  but  does  bear  close  monitoring. 

5.  Of  the  completed  PTRs,  57  percent  were  caused  by 
coding/design  errors.  21  percent  were  classified  as 
other  and  not  designated  to  any  category.  However, 
42  percent  of  the  critical  PTP.s  were  caused  by 
program  or  coding  errors.  Steps  have  been  taken  to 
better  divide  the  cause  category  in  an  attempt  to 
better  evaluate  tha  errors  that  were  classified  in 
the  other  category. 

б.  The  comparative  completion  rates  for  critical  PTBs 
has  remained  relatively  stable  compared  to  the  past 
year.  However,  tha  non-critical  completion  rate  has 
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increased  substantially  and  the  trend  is  fcr  ar.  even 
greater  completion  rate. 

7.  The  Receivad-Resolvad-Ouzstanding  results  show  zha- 
the  number  of  PTRs  received  is  leveling  off,  the 
number  of  PTRs  resolved  is  increasing  drastically, 
and  as  a  result  of  both  of  these  the  number  of 
outstanding  PTRs  is  now  in  a  steady  decline. 

8.  The  number  of  programs  released  increased  about  12 
percent  over  the  previous  quarter  and  yet  the  number 
of  PTRs  received  decreased  by  about  7  percent. 
Although  the  number  of  PTRs  over  the  last  year  has 
oscillated,  the  trend  shows  there  to  be  no  signifi¬ 
cant  increase  and  thus  a  net  decrease  in  ratio  of 
programs  to  PTRs  should  be  anticipated. 

Considering  that  only  nine  personnel  are  assigned  tc  the 
Quality  Control  Branch  and  that  FMSO  now  has  10,000  plus 
programs  in  existence,  the  Quality  Control/Quality  Assurance 
plan  appears  to  be  headed  in  the  right  direction  as  *he  ?TR 
report  clearly  shows.  With  continued  emphasis  on  the 
Quality  Control  effort  an  ever,  more  effective  program  will 
be  displayed  in  the  future.  To  achieve  a  100  percent  error 
free  product  is  the  ideal  but  a  more  realistic  goal  must  be 
set  and  an  effective  method  of  measuring  and  selecting  the 
programs  for  a  more  detailed  investigation  will  aid  greatly 
in  accomplishing  the  goal  FMS3  sets  for  itself.  The  PTR 
report  shows  an  improvement  over  the  proceeding  year  but  it 
also  quite  effectively  shows  other  areas  that  may  need  addi¬ 
tional  emphasis.  In  the  following  chapter  we  will  list  some 
of  the  areas  that  we  feel  should  be  considered  in  the  future 
corporate  growth  pattern  of  FMSO. 


107 


V.  RBC0HHSWDATI3SS 


This  paper  has  been  a  comprehensive  invesxigation  of 
software  quality  assurance  from  a  theoretical  viewpoint  and 
from  specific  investigation  into  the  quality  assurance/ 
quality  control  departments  of  various  organizations.  Based 
on  the  authors*  research,  the  following  recommendations  are 
offered  in  hopes  of  enhancing  the  Pleet  Material  Support 
Office's  (FMSO)  quality  control  effort. 

1.  A  corporate  base  line  concerning  tve  qualiry  orccass 
needs  to  be  established.  Before  a  specific  direction 
can  be  maintained,  a  mechanism  must  exist  that  would 
enable  FMSO  to  measure  its  ability  no  meet  the 
desired  objectives.  It  is  felt  that,  at  a  minimum, 
the  Quality  Control  Branch  could  examine  past 
programs  and  select  at  least  two  than  were  very 
similar  fcr  an  analysis  as  to  the  effectiveness  of 
the  quality  control  program.  The  criteria  of  These 
two  programs  might  be  (1)  that  they  were  produced  by 
the  same  department  within  a  short  time  period  oc 
each  other,  (2)  that  they  were  intended  to  be 
utilized  in  the  sane  fashion,  and  (3)  that  one  of 
them  had  been  reviewed  by  Quality  Control  throughout 
the  entire  process  and  the  other  had  not  received 
this  same  critical  review. 

2.  Top  management  must  continue  to  re-emphasize  the 
importance  of  quality  control  within  the  organization 
and  display  a  positive  philosophy  of  commitment  to 
the  quality  process.  ks  has  been  discussed,  without 
top  management  support,  quality  assurance/quality 
control  programs  produce  a  less  than  desirable 
output. 


3.  The  number  of  personnel  assigned  to  the  quality 
control  branch  is  definitely  inadequate.  I  r.  an 
organization  as  large  as  PMSO,  which  produces  300 
plus  programs  per  quarter,  a  quality  control  organi¬ 
zation  of  only  eight  people  plus  a  supervisor  can  not 
reasonably  be  expected  to  meet  their  obligations  and 
responsibilities  on  all  occasions.  The  Quality 
Control  Branch  needs  an  infusion  of  personnel. 
Attention  should  be  paid  not  only  to  the  quantity  of 
personnel  but  also  to  quality.  As  stated  previously, 
the  core  of  these  personnel  need  to  be  as  knowledge¬ 
able  as  senior  systems  analysts  and  also  command  the 
respect  of  the  individuals  whose  systems  are  being 
evaluated . 

4.  A  method  of  augmenting  the  staff  of  the  Quality 

Control  Branch  from  an  external  source  may  be  to 
utilize  it  as  an  indoctrination  facility  for  new 
hires.  A  core  of  highly  qualified  personnel  could  be 
maintained  that  were  the  permanent  part  of  the 
Quality  Control  Branch.  Hew  hires  could  be  given 
this  temporary  position  and  tasked  with  such  projects 
as  reviewing  the  project  specifications,  all  of  the 
manuals,  and  all  other  applicable  material.  This 
would  also  afford  the  new  personnel  an  opportunity  to 
receive  formal  and  correct  training  before  being 
assigned  to  the  particular  line  department. 

Additionally,  it  would  enable  the  new  hires  to  gain  a 
better  overall  understanding  of  what  the  organization 
does  and  the  amount  of  interfacing  that  must 

conducted  before  a  project  is  completed.  Of  course 
there  would  have  to  be  some  discretionary  measures 
imposed  when  selecting  personnel  and  a  time  limit 
must  be  adhered  to. 
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5.  Consideration  should  be  given.  ~o  moving  the  Quality 

Control  Branch  out  of  its  present  management  struc¬ 
ture.  The  authors  feel  that,  because  of  the  massive 
size  of  the  organization  and  enormous  amount  of 
material  that  must  be  reviewed,  the  Quality  Control 
Branch  should  be  moved  so  as  to  have  at  least  parity 
with  line  management.  This  position  may  be  desig¬ 
nated  as  Quality  Control  Department  Head,  Code  99. 
Billet  descriptions  would  have  to  be  re-written  and 
the  management  problems  overcome,  but  the  increased 
communications,  be  they  voluntary  or  by  direction, 
between  quality  control  and  line  management  would 
have  an  impact  upon  future  products.  Additionally, 
this  would  afford  the  quality  control  personnel  an 
easier  avenue  to  become  more  directly  involved  with 
their  responsibility  of  ensuring  that  standards  are 
met  and  better  a nable  them  to  enforce  these 

standards . 

6.  Line  managers  must  be  educated  as  to  the  importance 

of  the  quality  control  function.  In  this  vein,  wher. 

openings  arise  in  the  Quality  Control  3ranch,  quali¬ 
fied  line  personnel  should  be  encouraged  to  apply  for 
the  positions.  The  benefit  gained  for  the  organiza¬ 
tion  would  more  than  offset  the  loss  to  one 

individual  department.  To  support  the  attitudinal 
change  that  must  take  place,  an  ongoing  training 
program,  sponsored  by  top  management,  will  have  to  be 
implemented  on  a  company  vide  basis  and  the  positive 
benefits  gained  from  the  addition  of  these  qualified 
personnel  to  the  Quality  Control  Branch  must  be 
discussed  and  displayed. 

7.  Quality  control  checklists  must  be  re- instituted. 
There  should  be  a  general  checklist  applicable  to  all 
programming  functions  as  well  as  specific  checklists 
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regarding  each  individual  program.  I*  should  be 
emphasized  that  these  checklists  are  to  be  used  as 
reminders  or  as  a  constructive  tool  to  reinforce 
standards ,  rather  than  a  method  of  laying  blame. 

8.  It  is  important  for  continuity  that  there  be  consis¬ 
tent  assignment  of  quality  control  personnel  to 
projects.  One  person  {or  team  of  persons)  should  be 
assigned  to  a  project,  except  the  smallest  ones,  and 
should  monitor  this  project  all  the  way  from  incep¬ 
tion  to  post  implementation  review. 

9.  A  better  method,  possibly  a  decision  support  system, 
must  be  devised  co  determine  which  projects  will 
undergo  quality  control  procedures.  It  is  recognized 
that  the  number  of  projects  may  preclude  all  projects 
being  scrutinized  by  quality  control.  However,  there 
needs  to  be  a  better  system  to  select  programs  for 
review  and  evaluation  than  the  present  method  of 
committee  or  command  discretion.  Currently,  the 
primary  method  for  evaluating  project  length  and 
complexity  is  experience.  This  subjective  viewpoint 
is  the  basis  for  the  depth  of  involvement  achieved  by 
quality  control.  A  logical  and  comprehensive  deci¬ 
sion  process  will  serve  to  alleviate  crisis 


s  crisis 
important 


management  and  ensure  that  the  most  important 
projects  are  chosen. 

10.  End  users  must  become  even  more  involved  in  the 
design  and  development  of  software  throughout  the 
lifecycle.  Users  should  work  closely  with  quality 

control  to  ensure  a  timely  and  correct  review 
process.  There  _aust  also  be  better  communications 
established  between  users  and  line  programming  func¬ 
tions.  This  communication  may  be  facilitated  by 
quality  control  and  may  help  lower  project  trouble 
report  (PTR)  reclassifications. 
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11.  Clear  and  concise  documentation  is  an  ongoing  problem 
at  any  software  production  facility.  Jhe  quality 
control  division  must  review  documentation,  not 
coding,  and  assist  where  possible  to  iaprcve  any 
deficiencies. 

12.  Currently,  the  program  trouble  reports  (PIR)  ,  are  the 
primary  documented  measure  of  the  effectiveness  of 
the  guality  program.  While  this  may  be  a  valid 
measurement,  there  needs  to  be  a  search  for  addi¬ 
tional  measures.  Since  guality  control  does  not 
impact  100  percent  of  the  programs  leaving  FMSO,  a 
tiethcd  of  evaluating  the  effect  cf  guality  control  on 
projects  which  do  not  have  quality  control  involve¬ 
ment  must  be  formulated. 

13.  The  principle  of  top-down  design  and  tcp-dowr.  testing 
should  be  reevaluated.  As  an  alternative,  top-down 
design  and  bottom  up  testing  should  be  considered. 
It  is  assumed  that  top  managaaant  will  feel  uneasy  as 
the  present  process  is  changed.  However,  it  has  been 
shown  in  many  studies  that  errors  made  in  the  design 
phase  of  a  project  are  the  most  expensive  to  repair 
because  one  has  to  return  back  to  the  beginning  and 
literally  begin  the  entire  process  ever. 
Requirements  must  be  reevaluated,  specifications 
redone,  coding  rewritten,  and  finally,  the  program 
must  be  tested  again.  Since  the  majority  of  design 
errors  are  not  found  until  the  testing  phase,  it  is 
easy  to  see  that  the  amount  of  extra  time  spent  in 
the  design  phase  will,  over  time,  offset  any  anxiety 
that  top  management  would  have  because  of  che  seem¬ 
ingly  lack  of  progress  on  the  project.  It  is 
suggested  that  bottom-up  tasting  will  force  the 
design  effort  to  improve. 
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14.  Phased  development  should  be  adhered  to.  Coding  must 
not  begin  until  after  the  design  is  complete.  By 
doing  so  the  lead  programmer  ran  assign  the  better 
prcgrawers  to  the  difficult  aodules  and  an  inexperi¬ 
enced  prograaaer  to  the  easier  modules/programs. 
This  line  of  thought  should  filter  throughout  the 
project  and  allow  for  better  personnel  utilization, 
but  should  also  afford  the  persons  writing  the  test 
plan  to  do  so  in  a  lore  reliable  manner.  In  allowing 
the  writing  of  code  before  the  final  design  is 
completed,  •  we  achieve  the  short  term  goal  of  being 
able  to  see  a  working  product,  that  can  be  tracked  on 
a  chart.  However,  we  cannot  see  the  long  range  goal 
of  whether  the  nodule  will  produce  the  required 
output  or  the  required  number  of  interfaces. 

15.  The  system  to  trace  the  phase  of  development  for  each 
project  should  be  reemphasized  and  quality  control 
must  continue  to  be  kept  abreast  of  all  the  current 
production  efforts. 
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